Saturday, September 25, 2004
Statistics 588 - Data Mining: "I have further questions on Permutation Test: My understanding is to permute on the whole data set (500 in this case) to come up with 999 samplings without replacement, i.e. keeping the total number of injury accidents constant. Then my question is: whether apply GLRT to the whole data set with fixed window size to find smallest lamda in each sampling, then sort and find the position of the observed smallest lamda in this distribution; or apply GLRT only to the unusual windows observed from the orginal data set? Since there could be multiple unusual windows with the same smallest lamda calculated. If only applying GLRT to the unusual windows, then each unusual window needs to be tested for null hypothesis, right? J.C.
Basically whatever statistic you compute on the original data set, you compute exactly the same thing on each of the permuted datasets. So, if you are using a variable window size, you compute lambda for every possible window in the original datset and then find the smallest one. Then do exactly the same thing on each of the permuted datasets. So, you are not testing the particular unusual window that had the smallest lambda in the original data, rather you are testing if its particular value of smallest-lambda is unusual or not.
I'm a little concerned about the term "sampling with replacement" causing confusion here. You need to create random permutations of the labels of the accidents ("injury", "not-injury"). Sampling with replacement usually implies that some objects get sampled more than once and that is not what is happening here. D.
Basically whatever statistic you compute on the original data set, you compute exactly the same thing on each of the permuted datasets. So, if you are using a variable window size, you compute lambda for every possible window in the original datset and then find the smallest one. Then do exactly the same thing on each of the permuted datasets. So, you are not testing the particular unusual window that had the smallest lambda in the original data, rather you are testing if its particular value of smallest-lambda is unusual or not.
I'm a little concerned about the term "sampling with replacement" causing confusion here. You need to create random permutations of the labels of the accidents ("injury", "not-injury"). Sampling with replacement usually implies that some objects get sampled more than once and that is not what is happening here. D.
