Open in new window / Try shogun cloud
--- Log opened Mon Jul 09 00:00:17 2012
CIA-18shogun: Sergey Lisitsyn openmp * r4e109e3 / (2 files): Made CDistance use openmp -
CIA-18shogun: Sergey Lisitsyn openmp * r2b811d5 / (2 files): Made CombinedKernel use openmp -
CIA-18shogun: Sergey Lisitsyn openmp * r15203a3 / src/shogun/lib/external/shogun_libsvm.cpp : Made LibSVM use openmp -
CIA-18shogun: Sergey Lisitsyn openmp * r6cc5b15 / (2 files): Made DistanceMachine use openmp -
-!- n4nd0 [] has quit [Quit: leaving]00:47
-!- blackburn [~blackburn@] has quit [Quit: Leaving.]01:47
shogun-buildbotbuild #18 of nightly_default is complete: Failure [failed updatedoc]  Build details are at
-!- av3ngr [av3ngr@nat/redhat/x-xlpzesjskamqquhz] has joined #shogun04:11
-!- av3ngr [av3ngr@nat/redhat/x-xlpzesjskamqquhz] has left #shogun []04:12
-!- n4nd0 [] has joined #shogun09:38
-!- n4nd0 [] has quit [Ping timeout: 244 seconds]09:46
-!- uricamic [~uricamic@2001:718:2:1634:54ea:8a06:2501:ed7b] has joined #shogun09:50
-!- gsomix [~gsomix@] has joined #shogun09:56
gsomixgood morning09:56
gsomixsonney2k, got your mail. working on this problem now10:03
-!- blackburn [~blackburn@] has joined #shogun10:05
CIA-18shogun: Sergey Lisitsyn master * r47a4297 / src/shogun/mathematics/lapack.cpp : Made lapack use mallocs instead of SG_MALLOCs -
-!- blackburn [~blackburn@] has quit [Quit: Leaving.]10:44
-!- n4nd0 [] has joined #shogun10:49
-!- shogun-buildbot [] has quit [Quit: buildmaster reconfigured: bot disconnecting]10:57
-!- shogun-buildbot [] has joined #shogun10:57
@sonney2kgsomix, can you reproduce the problem? And do you think it is our fault?11:15
gsomixsonney2k, problems with trace mallocs are related to multithread. blackburn fixed already. but there is problem with CMap.11:17
gsomixand I think this is my fault.11:18
-!- pluskid [] has joined #shogun11:28
-!- pluskid [] has quit [Ping timeout: 264 seconds]11:39
-!- pluskid [~pluskid@] has joined #shogun11:52
-!- gsomix [~gsomix@] has quit [Read error: Connection reset by peer]11:54
shogun-buildbotbuild #19 of nightly_default is complete: Success [build successful]  Build details are at
-!- gsomix [~gsomix@] has joined #shogun12:10
gsomixsonney2k, around? it seems, that CMap works not properly with multithreaded insert/delete.13:18
gsomixsee you later guys13:18
-!- gsomix [~gsomix@] has quit [Ping timeout: 246 seconds]13:22
-!- gsomix [~gsomix@] has joined #shogun13:53
wikingn4nd0: ping14:05
n4nd0wiking: pang14:05
wikingn4nd0: pong ;)14:06
wikingok so question: svm^struct is going to be in ?14:06
n4nd0in like in shogun?14:06
n4nd0not exactly that implementation14:07
n4nd0what I am doing is a similar project though14:07
wikingsince currently u have mosek but i was wondering if u are planning to include svm^struct as well14:08
n4nd0no, I am not planning that14:08
n4nd0but we are planning to do something alternative that does not require mosek14:08
wikingah good that's good enough14:08
-!- pluskid [~pluskid@] has quit [Quit: Leaving]14:36
-!- n4nd0 [] has quit [Ping timeout: 244 seconds]15:12
-!- nietpiet [92322838@gateway/web/freenode/ip.] has joined #shogun15:19
-!- emrecelikten [~emre@] has joined #shogun15:45
-!- heiko [] has joined #shogun17:15
-!- gsomix [~gsomix@] has quit [Ping timeout: 246 seconds]17:35
-!- gsomix [~gsomix@] has joined #shogun17:35
-!- uricamic [~uricamic@2001:718:2:1634:54ea:8a06:2501:ed7b] has quit [Quit: Leaving.]17:43
-!- blackburn [~blackburn@] has joined #shogun18:07
blackburnsonney2k: sg_malloc is not thread-safe18:07
gsomixand also my CMap18:11
heikogsomix, do you have any idea about this trace-mallocs weirdness?18:13
blackburnheiko: thread-unsafetyy18:15
-!- n4nd0 [] has joined #shogun18:15
-!- cwidmer [8ca3fe9d@gateway/web/freenode/ip.] has quit [Ping timeout: 245 seconds]18:20
heikoblackburn, really? that is it?18:21
blackburnheiko: well we use cmap to store allocated blocks18:22
blackburnand insert is not thread safe18:22
heikobut that should be easy to fix18:22
heikojust lock whil inserting18:22
heikoblackburn, around?19:06
heikocomputing covariance matrices...19:06
heikocblas_dgemm ??19:06
heikoI saw that in your code19:07
blackburnwhat is the problem?19:07
heikowhat method is that?19:07
blackburndot product of matrices19:07
heikoah alright19:07
blackburnwell to be honest it is not covariance19:07
blackburnbut once we center it19:07
blackburnor center features before19:07
blackburnit is19:07
heikowhich way is best to compute cov(h) where h is a vector?19:08
heikolike the matlab equivalent19:08
heikothat way youre doing there?19:08
blackburndger I guess19:08
heikowhich is?19:08
heikoI am not used to all these libraries :)19:09
blackburnheiko: well what is cov(h)?19:09
blackburnx'*x resulting with matrix?19:09
blackburnif so cblas_dger does the job19:09
heikoempirical covariance matrix where the entries contains the covariance of pairs of the random variables in the input matrix19:11
heikowhich would be E[hh^T] - E[h]E[h]^T19:11
heikoand expectations are estimated by sample mean19:11
blackburnheiko: I'd suggest to cblas_dger and center_matrix after19:11
heikook alright19:12
heikoI will add a routine to CStatistics19:12
heikothis blas stuff is horrible to read ;)19:12
blackburnno if you are android19:12
blackburnI am19:12
heikolol :D19:12
heikoan evil robot, that what you are!19:13
heikowhat does dger do?19:13
blackburnheiko: just product of vectors19:13
heikoinner or outer?19:13
heikoresult is matrix or vecor?19:13
blackburngood shot19:13
heikook so I can do the hh^T thing with it19:14
blackburnbut matrix / scalar is the choice19:14
blackburnheiko: yes but do not forget to zero resulting matrix before19:14
heikooh man19:14
heikowill add that19:15
blackburnheiko: ^19:15
blackburncov_matrix += xx' that is19:15
blackburnor may be19:15
blackburncov_matrix += 1.0*xx'19:15
heikommh ok19:16
heikoIll dig into it19:16
blackburnheiko: if you are confused with ld* - it is just increment of dimensions19:16
blackburnso 1 for vectors and #num_of_rows for matrix19:17
heikobecause I still have no idea what to call ;) and I hate try and error code development19:28
blackburnheiko: what is wrong with dger?19:28
heikoits fine, I just hat these technical things, but Ill stop complaining now19:30
blackburnheiko: just ask me :)19:31
heikowell, if you know what to do, could you add a method static SGMatrix<T> cov(SGMatrix<T> observations) to CStatistics? :)19:31
blackburnheiko: sure19:32
blackburnheiko: observations is column-wise feature vectors?19:32
heikoit would be great if there was a bool flag for transposing19:32
blackburnheiko: transposing what? observations?19:32
heikoyes I mean if my data is the wrong way round it would be nice if I wouldnt have to transpose it before19:32
heikobut its not that important19:33
blackburnI just found19:33
blackburnin statistics19:33
heikoAh I see :)19:34
blackburnintegral looks very lovely19:34
heikohehe I might translate these into latex19:34
heikotaken from ALGLIB :)19:34
blackburnok ETA 20 minutes19:34
heikomany thanks :)19:35
heikoblackburn, they look horrible in doxygen documentation ;) I will definately change that19:40
-!- nietpiet [92322838@gateway/web/freenode/ip.] has quit [Quit: Page closed]19:41
blackburnheiko: can you test it afterwards?19:45
heikoI will :)19:45
blackburnhmm heiko it looks like in LLE I call kernel matrix as covariance matrix19:46
heikoah thats even better, I can copy it19:46
heikowhich lines?19:47
heikoblackburn, is it the one in 498?19:47
blackburnheiko: shogun/converter/LocallyLinearEmbedding.cpp:30919:48
heikothats dgemm19:48
blackburnheiko: yes19:48
blackburnheiko: linear kernel matrix that is19:49
@sonney2kblackburn, well then we have to make it so19:50
blackburnsonney2k: anyway lapacks mallocs are out of scope for tracing19:50
@sonney2kgoing back to just malloc will not solve the problem19:50
blackburnsonney2k: no but it shouldn't be tracked19:50
@sonney2kblackburn, but there is no error check in there so that won't do it19:51
@sonney2kblackburn, so what is not thread safe about them - the tracing stuff only right?19:53
gsomixwell time to work19:56
gsomixgood evening19:56
CIA-18shogun: Soeren Sonnenburg master * rda0a349 / src/shogun/mathematics/lapack.cpp : Revert "Made lapack use mallocs instead of SG_MALLOCs" -
@sonney2kgsomix, hi there - regarding thread safety of CMap - I guess all you would need to do is change the add / remove functions right?20:00
@sonney2kgsomix, if not other option would be to have a general acquire_lock() / release_lock() in CMap that one always has to call before doing sth with CMap20:01
heikoblackburn, finally got it :)20:08
@sonney2kwiking, don't miss your progress report this week!20:12
blackburnheiko: sorry was away for a while20:20
blackburnwill commit soon20:21
heikoI just found that I already did this20:21
heikoSGMatrix has matrix multiply20:21
-!- alexlovesdata [c25fae98@gateway/web/freenode/ip.] has joined #shogun20:21
heikousing cblas_dgemm20:21
blackburnheiko: okay should I do that then?20:21
heikosorry  :(20:21
blackburnno problem :D20:21
heikojust need to center matrix before then its fine20:21
blackburnheiko: after you mean?20:22
heikono before20:22
heikoand its not centering as center() method20:22
heikobut only remove mean of each column20:22
blackburnyeah exactly what I mean20:22
blackburnheiko: it is the same20:22
heikoare you sure?20:23
blackburnpretty much20:23
heikooh yeah20:23
heikojust seeing it20:23
blackburnbut it is more efficient20:23
heikoman, i should sleep more :)20:23
blackburnand more20:23
blackburnit does not modify matrix20:24
blackburnfeature one I mean20:24
heikonot modify?20:24
heikocenter() you mean()20:24
heikohow to not modify matrix?20:24
heikoone could do this with the +beta*A thing at the end of dgem20:25
blackburnheiko: well you don't have to subtract mean and to modify feature matrix20:25
blackburnonly covariance is to be modified20:25
blackburnyeah I think so20:26
blackburnlet me check20:26
heikoare you sure?20:26
heikono I think it has to be done before20:26
blackburnheiko: before what?20:26
heikobefore computing product20:26
heikoI mean one has to center feature matrix20:27
blackburnheiko: argh sorry20:27
blackburncenter works only with kernel matrix it seems20:27
heikoyeah there its fine20:28
blackburnheiko: just checked, yeah20:30
blackburnsad it doesn't work with covariance20:30
@sonney2kgsomix, why do you think that is slow?21:03
gsomixsonney2k, I had to do it at morning... :)21:06
@sonney2kgsomix, I mean basically none of the shogun stuff is thread safe21:06
@sonney2kwe make things thread safe when we need it and I am not so sure about CMap here21:07
blackburnsonney2k: why to make it explicit21:07
@sonney2kblackburn, you mean the locking/ unlocking?21:08
@sonney2kif it significantly slows down the 'normal' use case then we should make it explicit21:08
gsomixsonney2k, aww, just got your mail21:09
blackburnsonney2k: slows down significantly?? how21:09
blackburnlock is not that slow21:09
@sonney2kblackburn, did you measure?21:10
blackburnsonney2k: did you?21:10
blackburnI didn't21:11
@sonney2kblackburn, but basically no data structure or anything in shogun is thread safe21:11
blackburnsonney2k: right21:12
blackburnbut it should be actually I think :)21:12
@sonney2kwhile I don't mind adding thread safety for relevant cases we should do it only when really necessary21:12
@sonney2kblackburn, just think of someone calling train() of an SVM twice (in two threads...)21:13
@sonney2klots of stuff will just kaboom21:13
@sonney2kgsomix, can you quickly measure say how long it takes to add say 100000 items via add() to CMap?21:15
@sonney2kgsomix, w/ and w/o locks?21:15
blackburnsonney2k: evaluations are open now :)21:20
@sonney2kblackburn, is this good or bad?21:22
@sonney2kblackburn, btw21:22
blackburnsonney2k: gsoc one I mean21:22
@sonney2kgrep pthread shogun/ -r -l --include='*.cpp' | grep -v conv | wc -l21:22
@sonney2knot so many ...21:22
blackburnI know21:22
blackburnsonney2k: and 4 done already21:23
@sonney2kyeah problem is that this is not a trivial task and it is easy to make mistakes21:25
blackburnsonney2k: it is not that bad actually21:30
blackburnI just paste helper thing into the place where pthread code was21:31
alexlovesdata@sonney2k: I will take a deep look into Viktors Code tomorrow21:32
n4nd0sonney2k: around?21:35
@sonney2kblackburn, well look at shogun/features/DotFeatures.cpp21:36
@sonney2kactually that is not so difficult in this one21:36
@sonney2kblackburn, shogun/features/HashedWDFeaturesTransposed.cpp CHashedWDFeaturesTransposed::dense_dot_range_helper21:37
@sonney2kn4nd0, no :P21:37
n4nd0sonney2k: I have planned some holidays for the next week, so I'll be some days off work; is there a big problem with it?21:38
n4nd0sonney2k: I am certain it won't destroy the project's schedule21:39
@sonney2kn4nd0, np - just continue quickly when you are back21:40
n4nd0sonney2k: yeah, I have also reserved more time for this week21:40
n4nd0thank you21:40
n4nd0is there any way we get to know if our mentor has already sent the midterm evaluation?21:41
@sonney2kn4nd0, no21:42
-!- emrecelikten [~emre@] has quit [Quit: Leaving.]21:43
gsomixsonney2k, sorry. I'm back.21:44
gsomix>> quickly measure | is it actually needed?21:45
gsomix>> acquire_lock() / release_lock() | for lock/unlock? or for lock_init/lock_destroy?21:45
@sonney2kgsomix, what I had in mind is measuring CMap w/o locking21:47
@sonney2k(only add() function)21:47
@sonney2kand then w/ locking21:47
@sonney2k(from the patch you just sent)21:48
@sonney2kwe can assume spinlocks so it should be pretty fast21:48
@sonney2kbut lets see21:52
blackburnsonney2k: I am pretty sure locking of free lock is fast - no matter spinlock it is or mutex21:56
@sonney2kblackburn, my bet is that it is 10-100 times more expensive than x[i]=y21:58
blackburnsonney2k: my bet it is as fast :D21:59
@sonney2kgsomix, be the judge :D21:59
-!- zxtx [] has quit [Ping timeout: 248 seconds]21:59
@sonney2kn4nd0, nico just sent me an email that he did the evals22:01
blackburnsonney2k: well why do you think so?22:01
@sonney2kgut feeling :D22:02
@sonney2kwhy do you think it is fast?22:02
n4nd0sonney2k: yeah, thank you :)22:02
blackburndo you think any waits are here?22:02
blackburnit should be just atomic some bool=true22:02
blackburnwell I don't know how to do that atomic though22:02
-!- alexlovesdata [c25fae98@gateway/web/freenode/ip.] has quit [Quit: Page closed]22:03
@sonney2kn4nd0, btw before you leave just sent some early weekly report then - thanks22:09
@sonney2kblackburn, with spinlocks i think it really is just accessing some integer22:10
@sonney2kwith the pthread stuff it is sth more complex22:10
@sonney2kbut lets see22:10
@sonney2kgsomix, any news?22:10
blackburnwith pthread stuff?22:10
blackburnisn't spinlock pthread stuff?22:10
gsomixsonney2k, measuring22:10
@sonney2kgsomix, btw I cannot access melange and fill out your evaluation...22:11
gsomixsonney2k, hmm, I'm not sure that I did it correctly. but... code and results22:22
-!- ckwidmer [8ca3fe9d@gateway/web/freenode/ip.] has joined #shogun22:24
blackburnsonney2k: statistically insignificant22:24
gsomixoh, I had to make seven measurements... as said in the training manual for physics labs. :)22:26
gsomixI'm slowpoke... it seems that sonney2k gone to sleep22:28
@sonney2kgsomix, can you please measure only the add part with CTime and one more thing - are you using some on-demand cpu frequency scheduling?22:29
@sonney2kgsomix, turn that to a fixes freq in that case22:29
@sonney2kand for how to use CTime:  CTime t;  t.cur_time_diff(true);22:30
@sonney2kthat's it22:30
gsomix>> turn that to a fixes freq in that case | I don't know how. :(22:30
@sonney2kgsomix, cpufreq-set -r  -g performance22:36
blackburnheiko: ping22:43
gsomixsonney2k, blackburn
blackburnsonney2k: HAHA23:01
gsomixhm, weekly report23:03
@sonney2kgsomix, what is this print out? please use cur_time_diff() ...23:11
blackburnsonney2k: you looooost23:11
@sonney2kblackburn, c'mon the pthreaded stuff looks even faster than the other?23:11
blackburnsonney2k: INSIGNIFICANT :D23:12
gsomixsonney2k, okay. .____.23:12
@sonney2kgsomix, I think you used the clock_t based function...23:12
gsomixI used CTime with cur_runtime_diff23:13
@sonney2kblackburn, yes it seems to be fast enough though23:13
gsomixthat "get time difference between start and NOW"23:13
@sonney2kgsomix, runtime_diff is not cur_time_diff23:13
@sonney2kI guess we need to modify the function names23:13
@sonney2kone is measuring cpu time23:14
blackburnyeah cur_time_diff23:14
@sonney2kthe other wall clock time (in higher resolution)23:14
blackburnis *impossible* to remember23:14
@sonney2kblackburn, suggestions for a better name?23:14
gsomixoh, ok23:14
@sonney2kblackburn, gsomix it is like time <cmd>23:15
@sonney2kreal == wall clock time23:15
blackburnsonney2k: I do not have any actually23:15
@sonney2kuser == run_time in CTime23:15
blackburnmy methods have bad names too :D23:15
@sonney2kI am no longer sure that the runtime based functions are ok...23:16
@sonney2kblackburn, well for a start - how about get_wall_clock_time() ?23:17
@sonney2kand get_consumed_cpu_time() ?23:17
@sonney2kinstead of23:17
@sonney2kand get_runtime()23:17
@sonney2kbut I don't know how to name the functions measuring the difference between the saved start time and 'now'23:18
@sonney2kI like that23:19
heikoblackburn what was it?23:22
heikosorry was having diner :)23:22
blackburnheiko: no problem not actual I think23:22
blackburnI just didn't manage to find one xval function23:23
blackburnbut now did23:23
heikook :)23:25
CIA-18shogun: Heiko Strathmann master * r371587b / (9 files in 5 dirs): Merge pull request #634 from karlnapf/master (+8 more commits...) -
heikoblackburn, I just added the covariance :)23:26
heikowill log off now, good night23:26
-!- heiko [] has left #shogun []23:26
gsomixblackburn, sonney2k
@sonney2kgsomix, what is the code?23:35
blackburnsonney2k: does pycon fits to present shogun?23:38
gsomixneed to sleep23:43
gsomixgood night guys23:43
-!- gsomix [~gsomix@] has quit [Ping timeout: 244 seconds]23:49
--- Log closed Tue Jul 10 00:00:17 2012