There has been much discussion around this topic and after doing some extensive reading (and having been a student of stats) the "randomness" of LEED's guidelines fly in the face of statistical accuracy.
That being said, you have to live with what you've got. My questions are in reference to this same ideal.
In doing the “standing in front of the elevator methodology” to surveying, can you advertise that you will be conducting a LEED Transportation Survey or is that against the rules?
Does it matter what you give out as a teaser - for instance some people like bagels and coffee while others don't and some people don’t like cold weather so giving away a ski vacation might skew results???
One person said, "position a team at the elevators and sample every fourth person who enters - will get you the highest return, because you are much less likely to have non-respondents." I don't understand why sampling every fourth person makes sense...? In order to get the minimum sample respondents we use the formula - (occupants*752)/(occupants+752) in our case we have a sample of 626 (6 entrances and 10+ elevators…). So shouldn’t a team just wait at the elevator bank for a minimum of 4 hours until they have hit their sample size? Shouldn’t the team sample everyone? I understand that some people will not want to respond and therefor counted as SOV drivers as well as others your team just won’t be able to catch as they scoot by (assuming those are not counted at all) but the every four I was confused by. I also wonder about the if… What happens IF we don’t hit our sample after being at the elevator bank for 6+ hours? Must we give up and count the remaining portion of the sample size as SOV drivers?
Appreciate all the help – thanks LEEDuser team!
Jenny Carney
Vice PresidentWSP
LEEDuser Expert
657 thumbs up
May 20, 2010 - 12:44 pm
FYI, the "sampling" methodology is mostly derived from the SCAQMD guidelines (not a LEED invention). To try to hit on some of your other questions:
1. Advertising - at the time of the survey it's fine to make it clear what you are doing and why, but you shouldn't do things in advance of the survey that would skew occupants behavior from the norm or incentivize them to misreport their behaviors.
2. Teasers - if you are doing a random sample approach, you need to determine who your targets are independent of giving them a prize. For example, you can set up a free ski trip booth and wait to see who wanders up. You could, though, approach every 10th person and say "I have a survey, and if you are willing to take it I will give you a free ski trip." In this second case, the ski trip may influence whether they answer your questions but, importantly, doesn't influence who is included in the sample population.
3. Sampling vs. asking everyone - if you ask everyone, you are by definition no longer using the sampling approach. I think the suggest to ask every fourth person was just an example of a sampling technique that works, not a declaration that focusing on every forth person is a magical strategy that will get you the highest response rate. I've seen teams ask every next person that comes through the door, which is fine too. You have to have some strategy though so you're not just zeroing in on people wearing bike helmets or stepping off the train, though.
Matthew Macko
PrincipalEnvironmental Building Strategies
66 thumbs up
May 20, 2010 - 1:33 pm
Thanks for your answers
I know that LEED defines very little but uses known standards as reference points. I've also read the entire SCAQMD's multiple guidelines and what I've found in both reading those and talking to them is that the methodology is flawed because you can't both ask for a sample and set sample parameters (i.e. minimum sample size and conditions length of survey).
I get the advertising response - thank you - however I think you meant to say "can't" in regards to the free ski trip booth in the Teasers section... Let me know - a little confusing.
About the sampling - If you ask everyone or every next person, those are virtually the same thing - in fact what is to say that asking every 10 of 11 people would be using a sampling approach. I really want to get at the heart of the issue here which is that asking everyone vs. asking every fourth uses the same approach. In a big building with volumes of people there is no way you are going to get everyone so inherently you aren't asking everyone but you are doing what you can to get responses from the audience as they pass by in haste. The people you ask will be random because on any given day, who walk through the door next is completely random. The problem lies in the guidelines to get the sampling number. The 752 number seems arbitrary and at the end of the day no statistical z score or t score statistical extrapolation of data will be used to understand the population accurately. It makes sense to set rules because you don't want to have teams just asking sweaty bikers. I'll probably end up moving forward with doing what we can to not skew by staying there until we get our sample (to avoid losing non respondents to SOV) and somehow ensuring we get a sample as opposed to a sample that looks exactly like the population...
Dan Ackerstein
PrincipalAckerstein Sustainability, LLC
LEEDuser Expert
819 thumbs up
August 5, 2010 - 11:42 pm
Matthew - I think you've accurately identified some of the compromises USGBC necessarily had to make in balancing the statistical validity of the survey with the realities of asking non-expert facility folks to conduct a survey of this sort. There's a perfect statistical approach, there's an easy painless survey approach, and somewhere in between is where we tried to land. To your points - the key to the 'advertising' issue is creating response bias. If you advertise the fact that you're going to conduct a survey next week, you influence behavior in favor of the surveyed activity. In the same way, elsewhere I saw it noted that the elevator approach would eliminate nonrespondents - this is only true if the few people who refuse to answer survey questions (I've performed elevator surveys at a number of buildings, and there are always folks who just don't want to talk to you) don't know the subject of the survey. The minute you tell them its a survey about LEED or alternative transportation, you've created a response bias and therefore have to count 'refusers' as nonrespondents. Alternately, you can ask them if they'll take a survey without noting the subject matter, and if they say 'no' you don't have to include them as there is no bias inherent in their response. As to your question about every fourth person vs every person - I agree that its arbitrary. I think its fine to ask every person entering because arrival time at the building is as randomizing a variable as any other. One might argue that in some buildings there is a demographic element to arrival time and its something to be consider, but in most buildings its not meaningful. My advice is to show up early and fill out surveys until you hit your magic number, then do a few more for good measure. And on that note, I can tell you that the 752 figure is by no means arbitrary; its based on establishing a confidence interval that the USGBC felt comfortable with in terms of extrapolating data to the whole building population. We certainly could have done more in this regard, but again, we didn't want this to be an exercise in statistical analysis - just a manageable one in information gathering. Hope that helps - good luck with your survey.