Statistics 4240 - Data Mining

There are some excellent project ideas here.

This might be of interest to y'all. D.


Ansaf Salleb-Aouissi to Learningcenter, Tristan, Ruiyang, Gene, Haoyun
show details 11:32 AM (1 hour ago)


Hi everyone,

I will be giving 2 lectures this week and next week at Tony’s advanced ML class.
Just in case you are interested in the topic you’re very welcome to attend.
The class is COMS 6772 Advanced Machine Learning & Perception on Tue 4:10-6:00 at 1024 Mudd.

Tuesday Oct 28th – Intro to Frequent patterns (FP) and association rules (AR), definitions, examples, Apriori Algorithm: Mining FP and generation of AR, algorithmic challenges, AR as a probabilistic framework, evaluation measures, etc

Tuesday Nov 4th - Apriori variants, maximal FP, closed FP, quantitative FP, classification AR, current issues etc. (note: this lecture might be postponed to the 5th because of the election day).

Best,
Ansaf

_______________________________________________
learningcenter mailing list
learningcenter@lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/learningcenter

Does R have a for. . .each loop or just for loops?

Just a for loop as far as I know

Also, is there a way to get the index of a certain value in a vector/array/matrix/etc? For example you have vbl <- c(3,5,7,2,6) and you want to figure out what position 7 is in.

Check out the "which" function:

a<-c(1,2,3,4,5,6,9,8)
which(a==9)
[1] 7

Question about the final:
I am enjoying the format of the class - we get to hear about a lot of interesting things going on in data mining, without taking tons of time to get bogged down with the mechanics of every method, and then we get to choose what we want to learn more about. That said, I am wondering what the exam will cover and at what depth? I feel much more comfortable with the homework and project, where we work with actual data and get to see how different methods perform. To me, that shows more about what I've learned from the class than whether I can answer questions about techniques that were glossed over really quickly...

I certainly won't ask detailed questions about material we skimmed over. I'd like to ask questions from the book but I need to look at them a bit more carefully. D.

I just posted a third homework assignment. D.

A faculty at the Medical Campus is looking for Data Mining students interested in working with him to merge two datasets (state based and web based) to examine causes of Katrina-related deaths. Please email Professor Richard Garfield (rmg3@columbia.edu) if you are interested. Thank you!

I don't find any chapter related to text analysis in HTF book. So is there any books or papers about text categorization your recommend as reference? Thanks.

This paper by Fabrizio Sebastiani is a bit dated but provides an excellent overview.

This Thursday I will not be able to stick around after class. D.

In the example used in class for Naive Baye Prediction, I think the weights of evident of "present" and "absent" for the same X should add to 0 since the they are the log ratio of reciprocal numbers. But the slide shows that "0.51" and "-1.10" is a pair of "present" and "absent" weights of evidence. I don't know why...

One weight of evidence is, for instance, log(p1/p2), while the other is log((1-p1)/(1-p2)). These are not reciprocals of each other and will not
add to zero in general.

Compared to the actural value of the "missing" data in Homework2, with my result, it turns out that the linear model gives the smallest RMSE (39.97). Ridge and Lasso gives 42.83 and 43.27 respectively( I only run the 10-fold cross validation once to choose the lamda/bound) . Is it reasonable that linear regression is better than ridge and lasso in some case?

Usually ridge and lasso are better than plain linear, but in a real application, anything can happen!

Omar spotted this article related to what we are talking about in class.

Here are the actual values for the 33 missing "newAccounts" in Homework 2.
Let me know what your root mean square error is. D.

155
244
266
368
326
174
462
304
289
265
286
7
324
207
161
233
288
524
265
192
623
351
221
82
526
161
235
80
334
288
122
766
101

In the graphs of cross-validation results for linear regression in the powerpoint slides and in the textbook, the x-axis is labeled "subset size".  I think that was just for an example where CV was being used for subset selection, correct?  That is not relevant to what we're doing for part 2 of the homework, right?

To find the least squares estimate of a regression line you need to
solve an optimization problem. With just one predictor variable you do this
handily without using matrix notation. For more than one predictor, you
sort of have to use matrix notation. The core calculation involves finding
the inverse of the product of the transpose of the X matrix with the X matrix
itself. That inverse does not exist when p > N (its because the product is
not of full rank).

Here is some code to do ridge predictions. Basically I add a column of one's and delete the newAccounts column before multiplication with the coefficient vector. If someone has a better way to do this, please post it. D.

foo<-read.table("http://www.stat.columbia.edu/~madigan/DM08/BankSortedMissing.TXT",header=TRUE)

bankTrain<-foo[1:200,]
bankTest<-foo[201:233,]
nTest <- dim(bankTest)[1]

mylm<-lm.ridge(newAccounts ~ .,data=bankTrain,lambda=10)
myPred<-as.matrix(cbind(rep(1,nTest),subset(bankTest,select=-newAccounts))) %*% coef(mylm)

If I am extracting outliers to build the model, should I also extract them in the cross validation process?

You probably should but you could try it both ways. D.

In lm.ridge, it returned to me coeffiencts like this (partial):
numHouseholds numSalesStaff Manhatten 33.414153213 -0.011093286 16.773828208 33.688333432
the intercept has no name.
In the prediction step, we have to write down the fomula. how could we take the intercept out and put it into the fomula? I tried myridge$Inter, but failed.

I'll post something on this in a few minutes.

another problem is, the CV error (estimated by MSE) is depending on how you divide the groups, so it actually varies each time you run the program. this leads to a different best lamda (although they are close to each other) each time.

CV is a stochastic procedure so indeed the results are not entirely predictable.
You can run it multiple times and average the results if the instability becomes
a big issue. D.

So we are not required to do anything using lasso for the HW?

That's correct unless you want to try it for the optional part. D.

I am getting an error in R:
Error: could not find function "lm.ridge"

Am I missing something?

Try library(MASS) before calling lm.ridge. D.

Here are some links to large scale logistic regression tools:

http://www.stat.rutgers.edu/~madigan/BBR

http://www.stanford.edu/~boyd/l1_logreg/

http://www-stat.stanford.edu/~jhf/ftp/GPSpaper.pdf (not sure where the software is)


D.

anyone know how to add the std error bars to a plot in R??

As we discussed in the class, check out
tables2graphs

Here is my code for homework 1. D.


# read the data

myData <- read.table("/Users/dbmad/Documents/DM08/fatality.txt",header=TRUE)
x <- myData[,1]
y <- myData[,2]



# constants

N <- dim(myData)[1]
MAXWINDOWSIZE <- 250
MINWINDOWSIZE <- 5
NUMITER <- 100

# initialization

pValue <- 0;
smallestLambdaArray <- rep(0,NUMITER);

count <- 1 #first iteration of the algorithm is to find actual clusters, all other permutations are for comparison
compareprob <- rep(0, NUMITER)

while(count <= NUMITER) {
cat("iteration: ",count,"\n");
smallestLambda <- 100000
windowSize <- MINWINDOWSIZE
while(windowSize < MAXWINDOWSIZE){
windowStart <- 1
while (windowStart <= (N - windowSize + 1)) {
window <- y[windowStart:(windowStart+windowSize-1)]
positives <- sum(window)
negatives <- windowSize - positives
outWindowSize <- N - windowSize
outPositives <- sum(y) - positives
outNegatives <- sum(1-y) - negatives
if ((positives > 0) && (negatives > 0) && ((positives/windowSize)>(sum(y)/N))) {
windowLambda <- -1 * (log((positives/windowSize)^positives) + log((negatives/windowSize)^negatives) + log((outPositives/outWindowSize)^outPositives) + log((outNegatives/outWindowSize)^outNegatives))
if (windowLambda < smallestLambda) {
smallestLambda <- windowLambda
# cat(x[windowStart],"-",x[windowStart+windowSize-1],"\t",windowLambda,"\t",windowSize,"\t",positives,"\n");
if(count == 1) {
windowSizeOfCluster <- windowSize
clusterWindowStart <- windowStart
}
}
}
windowStart <- windowStart + 1
}
windowSize <- windowSize + 1
}
y <- sample(y, N, replace = FALSE)
smallestLambdaArray[count] <- smallestLambda
if (smallestLambdaArray[count] <= smallestLambdaArray[1]) {
pValue <- pValue + 1;

}

count <- count + 1

}

cat("Size: ", windowSizeOfCluster,"\n")

cat("Window: ",x[clusterWindowStart],"-",x[clusterWindowStart+windowSizeOfCluster-1],"\n")

cat("pValue: ",pValue/NUMITER,"\n")

i found another question. the RMSE is always increasing as the ridge constant goes larger(i.e. more restrict on beta's), as is shown in SAS. Here RMSE is generated from all observations, not some certain folds.

but i guess the result is the same if we get RMSE by Cross-Validation on some folds, cuz after trying many lambda's, i found that the best lambda is always the one before last candidate bounds, using the code you give. acutually the cvRMSE is always decreasing as bound goes to 1. this is for sure: if you put more constraints on beta's, you lose some accuracy on prediction.

so the question is: how do we find the BEST ridge if the cvRMSE is monotonely increasing as ridge constant goes larger?

thank you!

When I run l1ce with the bank data, cvRMSE selects a lambda that
is not the largest...

why can't i use predict function when i use the lm.ridge in R? thank you!

It appears that the predict function for lm.ridge was simply never implemented.