Researchers and applied practitioners try to minimize errors whenever possible. When conducting survey research, there are four primary areas where error most likely occurs: coverage error, sampling error, measurement error, and nonresponse error.

• Coverage error refers to an incomplete list of the population. For example, if you are looking to survey households in a community but your list only has homeowners, you will have an issue with leaving out renters.
• Sampling error refers to unintentionally missing a subset of the population. If your final sample does not reflect the population you will increase the likelihood of sampling error.
• Measurement error is when the observed value does not reflect the actual value. In other words, the answer you received from a survey questionnaire either does not represent the actual answer within the population, or the question you asked does not measure what you claim it does. The former is a problem tied to sampling and the latter is connected to the intent and design of the question.
• Nonresponse error occurs when certain segments of the population are less likely to complete the survey or sections of the survey.

Conceptualization refers to defining and specifying what a researcher or practitioner means by a specific term or word. Articulating what one means by a given concept helps establish the purpose of a research project and clarifies what is hoping to be measured and eventually understood. This process allows outsiders to see what the researcher is doing and how they are thinking about the central concepts.

Population refers to the entire group of something (e.g., all people in a community, all shoppers downtown, all business owners in the county).

Sample is a specific group selected from the population and the group that will complete the survey questionnaire.

Sampling frame is the list of the population from which the sample will be drawn. The ideal sampling frame is identical to the population.

Sampling design refers to the strategy or plans to identify those selected for the sample. Sampling designs fall into two camps, probability and non-probability-based sampling designs.

• Probability-based sampling is defined by elements of the sample having an equal chance of selection resulting in representativeness and generalizability.
• Non-probability-based sampling does not guarantee an equal chance of selection to elements of the population and therefore is not representative nor generalizable.

Representativeness refers to the sample reflecting the same characteristics and variations as the broader population.

Generalizability refers to the confidence one has to make statements about the population based on findings from the sample.

Operationalization is an extension of conceptualization. It refers to moving from the abstract to the specific and measurable. In other words, you are taking ideas and trying to find a way to measure and capture that idea in the real world through empirical observation (i.e., a survey question).

Validity refers to the accuracy of a question to capture the meaning of a concept. If you want to understand shopping behaviors downtown, you need to ask questions such as how often do you buy certain goods, from where do you buy them, how much do you buy, and so on.

Reliability is related to the consistency in obtaining the same results at different times. For example, if you are interested in shopping behaviors, you will want to ask questions that allow the respondent to answer the same each time. You would want to ask “how much milk did you buy last week?” versus “how much milk did you buy last year?”. The former question is much easier for the respondent to recall and to consistently respond with a dependable answer.

Business visitation surveys are a technique in which two-member teams visit all downtown business owners and ask questions. Use business visitations if you want to collect detailed business information and perspectives. Visitations also leave the owner feeling the most appreciated.

Although not officially a database, Microsoft Excel makes it easy to organize and analyze your data. One way to move your survey questionnaire into Excel is to establish a codebook, or a document that clearly shows what numerical values mean in your “database” and explain the detailed description of variable names. For example, if you ask the question matrix:

You would want to turn this survey question into a readable codebook item. The first thing you will notice is that this is not one but three questions (i.e., breakfast, lunch, and dinner). As such, you will want to create a short variable name. Next, you will need to assign a numerical value to the attributes (i.e., the options the respondents can choose). Finally, you will need to decide how you want to deal with missing data, meaning the respondent did not answer the question. Here is how you turn this question matrix into an item in your codebook:

When deciding what value to assign, you have a couple of things to keep in mind. First, you want to be sure to keep scales going in the same direction. For example, if you decide that the more often people eat out should have a “larger” number assigned to it, you will want to do the same for all questions throughout your survey. Maybe your survey includes a question about the “quality of existing restaurants in town” rated on a scale of “excellent” (=5) “good” (=4), “fair” (=3), “poor” (=2), and “terrible” (=1). To minimize headaches and work later and to allow for more sophisticated analysis, you will want to make sure to assign higher numbers to the more positive ratings.

Finally, do not forget to include a variable in your codebook for the respondent. This will sync with your “database” and provide easy access to check and clean data. Here is an example of how to set this up:

Once you have completed this process for the entire survey questionnaire, you can set up your “database” and begin data entry.

Basic data analysis includes simple univariate analysis and bivariate analysis. Univariate analyses examine a single variable within a dataset and are used to describe the characteristics of the sample. There are several ways to evaluate a single variable including measures of distribution, central tendency, and dispersion.

Frequency distributions refer to showing response by count and percent per variable attribute. For example, if you have asked 1,400 respondents to answer the question, “How often do you eat out for breakfast?” on a scale from “never” to “5 or more times a week”. You can examine responses in a frequency distribution table:

Measures of central tendency include mean (average), median (middle point), and mode (most common). The mean is calculated by adding all values of all responses and dividing it by the total number of responses. The median is calculated by taking the middle two numbers, adding them together, and dividing this number by 2. The number you get is the 50th percentile, suggesting that half of the responses fall below and half fall above. Finally, mode is simply the most common response and can be found by reviewing frequency distributions.

Dispersion is a way to understand how responses are distributed around a central point such as the average or mean. The simplest measure of dispersion is range, which is the difference between the lowest and highest value within the sample. For example, if your youngest respondent is 18 years old and the oldest is 88, the range is 70. Common measures of dispersion that require more sophisticated analysis and interpretation are variance (i.e., a statistic that provides insight into how wide the spread is from the mean) and standard deviation (i.e., a statistic that provides insight into the variability in a dataset as it relates to a normal, or bell-shaped, curve).

Bivariate (and multivariate) analysis starts the process of examining how variables relate to one another. The simplest method is to place data in a table that shows how one variable is distributed by the responses of another variable. For example, if you wanted to examine how the frequency of eating breakfast out is distributed by age, you could set up the following table:

*note, missing data has been removed from the analysis and percentages are calculated within the age category.

One can start to understand a lot about the sample by using cross-tabulated tables such as the one above. It will show if and where there are major differences by attribute. There are more advanced analyses that will further explore the associations between variables and can help answer questions such as do these two variables seem to increase or decrease with one another? What is the strength of the relationship between these two variables, if any? Common advance analyses include correlations (i.e., a measure of association between two variables), Chi-square (i.e., a test of variable dependence/independence), t-test (i.e., a specific test between the mean score for two groupings/attributes – e.g., is the average amount of times “younger” respondents eat out meaningfully different than the average amount of times “older” respondents eat out), and One-way ANOVA (i.e., test that examines the difference among the mean of three or more groupings/attributes).

The 2022 update of the toolbox marks over two decades of change in our small city downtowns. It is designed to be a resource to help communities work with their Extension educator, consultant, or on their own to collect data, evaluate opportunities, and develop strategies to become a stronger economic and social center. It is a teaching tool to help build local capacity to make more informed decisions.

This free online resource has been developed and updated by over 100 university educators and graduate students from the University of Wisconsin – Madison, Division of Extension, the University of Minnesota Extension, the Ohio State University Extension, and Michigan State University – Extension. Other downtown and community development professionals have also contributed to its content.

The toolbox is aligned with the principles of the National Main Street Center. The Wisconsin Main Street Program was a key partner in the development of the initial release of the toolbox. One of the purposes of the toolbox has been to expand the examination of downtowns by involving university educators and researchers from a broad variety of perspectives.

The current contributors to each section are identified by name and email at the beginning of each section. For more information or to discuss a particular topic, contact us.