Open in new window / Try shogun cloud
--- Log opened Thu Aug 30 00:00:17 2012
-!- gsomix [~gsomix@] has quit [Ping timeout: 256 seconds]00:03
-!- zxtx [] has quit [Ping timeout: 268 seconds]03:54
-!- zxtx [] has joined #shogun04:08
-!- zxtx [] has quit [Remote host closed the connection]04:15
-!- zxtx [] has joined #shogun04:25
-!- n4nd0 [] has joined #shogun07:09
n4nd0hi naywhayare07:09
n4nd0I read in logs you were asking about KNN yeseterday07:10
-!- blackburn [5bdfb203@gateway/web/freenode/ip.] has joined #shogun07:38
blackburnoh matplotlib's author john hunter passed away07:39
-!- n4nd0 [] has quit [Quit: leaving]08:05
-!- gsomix [~gsomix@] has joined #shogun08:46
-!- gsomix [~gsomix@] has quit [Quit: Ex-Chat]08:52
-!- gsomix [~gsomix@] has joined #shogun08:52
CIA-52shogun: Evgeniy Andreev master * r037f1ed / (114 files): many fixes in examples -
CIA-52shogun: Evgeniy Andreev master * r4cac666 / (204 files in 2 dirs): added env path -
CIA-52shogun: Sergey Lisitsyn master * r41d2350 / (207 files in 2 dirs): Merge pull request #761 from gsomix/examples -
@sonney2kblackburn, what?08:58
@sonney2kwhat happened to john?08:58
blackburnsonney2k: he passed away after cancer treatment08:58
gsomixgood morning08:59
blackburnhe was diagnosed with cancer in July and there were some complications during treatment08:59
@sonney2kblackburn, horrible09:01
blackburnyeah it is09:01
@sonney2kblackburn, I knew him in person09:01
@sonney2kwe organized an mloss workshop and invited him09:01
@sonney2kthere is this nice video on videolectures from him showing the power of matplotlib09:01
@sonney2kI even think he was subscribed to the shogun mailinglist09:02
blackburnyeah OSS is kind of big family and I am saddened09:02
-!- n4nd0 [] has joined #shogun09:04
@sonney2kblackburn, yeah he is still subscribed :/09:08
-!- n4nd0 [] has quit [Quit: leaving]09:19
shogun-buildbotbuild #485 of deb3 - modular_interfaces is complete: Failure [failed test python_modular]  Build details are at  blamelist: Evgeniy Andreev <>09:27
gsomixshogun-buildbot, aha, next commit, please :)09:28
CIA-52shogun: Evgeniy Andreev master * r148781d / (2 files): fixed order of imports -
CIA-52shogun: Sergey Lisitsyn master * rd321e36 / (2 files): Merge pull request #762 from gsomix/fix_examples -
shogun-buildbotbuild #486 of deb3 - modular_interfaces is complete: Failure [failed test python_modular]  Build details are at  blamelist: Evgeniy Andreev <>, Sergey Lisitsyn <>09:59
shogun-buildbotbuild #487 of deb3 - modular_interfaces is complete: Success [build successful]  Build details are at
gsomixshogun-buildbot, good work!10:39
shogun-buildbotWhat you say!10:39
-!- hoijui [] has joined #shogun12:07
-!- av3ngr [] has joined #shogun12:24
-!- av3ngr [] has left #shogun []12:24
-!- n4nd0 [] has joined #shogun13:10
-!- blackburn [5bdfb203@gateway/web/freenode/ip.] has quit [Ping timeout: 245 seconds]13:42
-!- blackburn [5bdfb203@gateway/web/freenode/ip.] has joined #shogun13:46
-!- gsomix [~gsomix@] has quit [Quit: Ex-Chat]15:20
@sonney2kn4nd0, do you know when bern4rd has time?16:47
@sonney2kor will be around?16:47
n4nd0sonney2k: I think he finished his internship yesterday16:49
n4nd0so probably he is coming back to Spain today, if he didn't do so yesterday16:49
n4nd0apart from that, I know nothing else16:49
n4nd0sonney2k: do you want me to ask him something?16:50
-!- in3xes [~in3xes@] has joined #shogun16:54
-!- in3xes [~in3xes@] has quit [Ping timeout: 244 seconds]17:01
-!- in3xes [~in3xes@] has joined #shogun17:11
naywhayaren4nd0: are you Dr. KNN for shogun?17:42
n4nd0naywhayare: haha not a Dr. :P17:42
n4nd0but I put the cover tree there and test some other things so I may be able to help you17:42
naywhayareokay; my goal is to find the distances and indices of the k nearest neighbors of a single dataset as simply and quickly as possible17:43
naywhayaresonney2k suggested I build the tree by hand17:44
n4nd0I recommend you to use JL covertree17:44
n4nd0there is no fastest algorithm for that17:44
naywhayareI mean, the fastest way to do it with shogun's available code specifically17:44
naywhayareJL's code does that, definitely, but his code is an unmaintainable nightmare17:44
naywhayareI do not have curse words descriptive enough for what I think of that work17:45
n4nd0hehe it takes time but I finally managed to use it in shogun17:45
naywhayareare you calling out to it, or did you port it?17:45
n4nd0I call it17:45
n4nd0I think it is the best ... I mean that code there has crazy stuff for optimization...17:45
n4nd0that's why it is so fast17:46
naywhayaremmm, yes and no17:46
naywhayareI don't like the CFLAGS he uses17:46
naywhayare-ffast-math is technically unsafe in some cases though it does give a 3x speedup (on the trials I ran)17:46
naywhayarewhen you take that away, the gains from the partial distance computation and the absurd stack-of-already-allocated-memory are not incredibly huge17:46
n4nd0you can just not compile with that?17:47
naywhayareyeah, modify the Makefile17:47
n4nd0how large is your dataset and what dimension does your data have?17:47
naywhayarethere are many datasets ranging from 100x3 to millions x hundreds17:47
n4nd0and maybe if you tell me a bit more of your application we can discover what is best to do using shogun17:47
naywhayarewell, I am using shogun to time it for comparison17:48
naywhayareI would prefer to use kd-trees to keep the algorithms the same across all the libraries I am comparing17:48
n4nd0we don't have kd-trees here I believe17:48
naywhayarehowever I didn't see any kd-tree support anytmore17:48
naywhayaresonne suggested that they did exist17:48
-!- in3xes [~in3xes@] has quit [Ping timeout: 240 seconds]17:49
n4nd0I think we don't have them17:49
naywhayareI need to step out for a little while... I'll be back later17:51
naywhayarethanks for the pointers so far17:51
n4nd0all right17:51
n4nd0let me know later17:51
n4nd0in any case, covertree is faster than kd-tree, AFAIK17:51
-!- blackburn1 [~blackburn@] has joined #shogun18:14
-!- in3xes [~in3xes@] has joined #shogun18:18
CIA-52shogun: Sergey Lisitsyn master * r4c43de1 / src/shogun/mathematics/Statistics.h : Fixed doc mistake in statistics -
blackburn1n4nd0: around?18:27
n4nd0tell me18:28
blackburn1n4nd0: wanted to think out loud :D18:28
blackburn1n4nd0: about serialization18:29
blackburn1okay the case is18:29
blackburn1we have a machine (say linear)18:29
blackburn1with w = vector of length 20018:29
blackburn1we use it somehow and then load from file18:30
blackburn1with different w length18:30
blackburn1I am confused what is happening there and what to do18:31
blackburn1first of all we corrupt SGVector18:31
-!- puffin444 [180317bb@gateway/web/freenode/ip.] has joined #shogun18:31
n4nd0in terms of the ref counter?18:31
blackburn1it is a different vector already18:32
blackburn1so we should create new reference counter I guess18:32
puffin444hey blackburn18:32
n4nd0but should it be a different vector?18:32
blackburn1puffin444: hey, any success with your issue?18:32
puffin444Yes in fact.18:32
blackburn1n4nd0: in simplest case "new machine then load" it is a different vector18:32
n4nd0blackburn1: I am assuming the vector you load from file is into the first vector you were talking about18:33
n4nd0yeah sure, in that case it ise18:33
puffin444It runs fine now. The only issue is that using the SGMatrices results in less precision.18:33
blackburn1yes but it is not true in general18:33
blackburn1puffin444: why?18:33
puffin444I really am not sure. It's the brent minimzer which seems to be where the precision is a little off.18:34
blackburn1but what is different with eigen matrices?18:34
puffin444All the calculations appear to be correct, is just that small numerical differences appear to result in an answer which is just slightly off that of using purely eigen matrices.18:34
blackburn1but still seems so strange18:35
puffin444It is absolutely essential that there are no eigen references in the headers, correct?18:35
blackburn1yes for now18:35
CIA-52shogun: Sergey Lisitsyn master * r80fee54 / src/shogun/lib/slep/slep_solver.cpp : Fixed memleak in slep solver -
puffin444Yes it does seem strange I took 4 hours to see what was wrong.18:35
blackburn1I can imagine18:35
puffin444And it just seems that the difference is due to numerical error.18:36
blackburn1how much is the error?18:36
puffin444The gradient search comes up with a mimimum likelihood of something like 5.35 instead of 5.34518:36
blackburn1huh that's pretty big18:37
puffin444Initially the two resuls are exactly the same but over time those numerical diffferences add up.18:37
puffin444Through gradient search.18:37
CIA-52shogun: Sergey Lisitsyn master * r78f693c / data : Updated data -
puffin444I can look at it more.18:38
puffin444It's just that I don't think I can get this in by Sept 1st18:38
blackburn1puffin444: we have some troubles with serialization actually18:38
puffin444Uh Oh - the hash function?18:39
puffin444Are you talking about my parameter hash function?18:39
blackburn1not really, there is a problem with reference counting and some more18:39
puffin444Oh okay.18:39
blackburn1I didn't detect anything with hash stuff18:39
puffin444Just to let you know I really want to make this part of a release for Shogun, and I hope to continue to contribute for the time being.18:40
puffin444I may be able to add more stuff as part of a class project this semester too, but I am not sure.18:40
blackburn1that's awesome to know because you are the only man that knows GPs here18:41
puffin444heh lol18:41
blackburn1I am totally out of GPs18:41
blackburn1puffin444: are you going to have masters study now?18:41
puffin444No it's actually Ph.D.18:41
blackburn1ahh so phd18:41
blackburn1I see18:41
puffin444It's probably going to focus on Computer Vision, but machine learning is part of everything here, so I will be definitely be working with it in some fashion.18:42
blackburn1I am hanging with CV as well so we could always discuss something18:43
puffin444Yeah sure :)18:43
-!- blackburn2 [~blackburn@] has joined #shogun18:45
-!- gsomix [~gsomix@] has joined #shogun18:45
gsomixgood evening18:45
blackburn2too many of me18:46
puffin444So I have to go now. Just wanted to give you an update on stuff. I'll see if I can get rid of the imprecision.18:46
-!- gsomix is now known as blackburn418:46
blackburn2puffin444: I don't really know whether we could release on 1st18:46
puffin444Okay. Are you guys thinking about pushing back the date?18:47
n4nd0puffin444: btw, did you find something strange in valgrind's trace at the end?18:47
-!- blackburn1 [~blackburn@] has quit [Ping timeout: 244 seconds]18:47
puffin444Oh valgrind detected a few read errors, which was the cause of the problem.18:48
puffin444There are still memory leaks though from Vector<CMap>.18:48
blackburn2puffin444: yes, I also had an idea of releasing RC (release candidate) first18:48
n4nd0I got angry some months ago too because my program didn't crash on valgrind but yes alone :D18:48
puffin444Okay good. I am working as much as I can to get this ready. I may be held back a bit by university stuff however. Am I holding you guys back?18:49
blackburn2puffin444: no, not really18:50
blackburn2we wanted a major version so better make it correct18:51
blackburn2I will try to handle serialization leaks next days18:51
blackburn2puffin444: anyway thanks :)18:52
puffin444Thank you for being understanding. I'll see what I can do to increase precision.18:53
n4nd0let's continue with serialization conversation?18:53
-!- puffin444 [180317bb@gateway/web/freenode/ip.] has quit [Quit: Page closed]18:54
blackburn2n4nd0: yeah18:57
blackburn2n4nd0: I am totally confused with reference counting18:57
blackburn2do you have an understanding?18:57
n4nd0the trivial one18:58
blackburn2okay let me check what is leaking19:00
blackburn2n4nd0: okay serialization_basic_tests19:00
-!- blackburn1 [~blackburn@] has joined #shogun19:02
blackburn1piece of honey, useful wired connection19:02
blackburn1n4nd0: here/19:03
-!- blackburn2 [~blackburn@] has quit [Ping timeout: 268 seconds]19:05
n4nd0I have no idea how these functions Parameter::Load work :S19:06
blackburn1n4nd0: I could try to describe19:06
blackburn1I can't say I like how it works but it is there :)19:06
n4nd0yeah sure19:07
blackburn1n4nd0: each time you call load/save_serializable19:07
blackburn1in basic case it iterates through all parameters19:07
blackburn1you set via SG_ADD19:07
blackburn1in case of vectors and matrices it stores pointer to pointer to data and pointer to lengths19:08
n4nd0I need a break, I've got sort of headache19:08
n4nd0haha not for this19:08
blackburn1okay sure19:08
n4nd0I have been reading on the computer for a long while before19:08
blackburn1do you have headache after that?19:08
blackburn1I can easily handle 14-16 hours using computer :D19:09
blackburn1you are not geek enough!19:09
n4nd0haha no, I don't think so :P19:09
n4nd0see you in a while19:09
blackburn1more geekness!19:09
-!- blackburn4 is now known as blackburn219:18
n4nd0all righty19:24
-!- in3xes [~in3xes@] has quit [Quit: Leaving]19:32
n4nd0do you think the use of m_parameters->add might be an issue?19:33
n4nd0I remember Heiko sent a mail saying to use SG_ADD19:33
n4nd0I think we could try to think what the state of the reference counters should be after each step of that test19:38
n4nd0e.g. how it should be after to_save->save_serializable(file)19:38
blackburn1n4nd0: okay leaking in serialization_basic_test is caused by wrong reference count after load19:38
blackburn1case is19:39
blackburn1SGObject -> DenseFeatures -> matrix19:39
blackburn1after reading SGObject matrix's refcount is -119:39
blackburn1however I have no idea why19:39
n4nd0is it correct after to_save->save ?19:40
blackburn1save does not corrupt19:41
n4nd0what is the ref count after save?19:41
blackburn12   when I get it19:42
n4nd0I don't understand what you mean with the last thing, sorry19:42
n4nd02 when you get it?19:42
blackburn1with get_feature_matrix19:42
n4nd0aham ok :)19:43
blackburn1yeah it was hard to parse probably19:43
n4nd0I will compile with refcount enabled19:43
n4nd0may be helpful19:43
blackburn1what do you mean with refcountenabled?19:44
n4nd0I didn't use the right name but I don't find in configure what I wanted to say19:45
blackburn1trace memory allocs?19:45
blackburn1no that won't really help19:45
n4nd0maybe it was that one yes19:45
blackburn1valgrind detects the error already19:45
n4nd0but I think it would be nice to see when the refcount is modified19:46
n4nd0it should get us to the root of the problem19:46
blackburn1yes that makes sense19:46
blackburn1n4nd0: it can be done with SG_GCDEBUG19:46
blackburn1no need to recompile19:46
n4nd0that was it :)19:47
n4nd0will we see something related to SGVector there as well? I remember I use it for SGObject19:47
blackburn1yes it is DEBUG_SGVECTOR and it requires to recompile19:47
n4nd0do you think it will be helpful?19:48
blackburn1yeah could be19:48
blackburn1I do not really understand how can it be -119:49
n4nd0DEBUG_SGVECTOR is an option in configure? don't find it19:50
blackburn1no you should put it manually to lib/config.h if you want19:50
n4nd0all right19:50
n4nd0compiling now19:52
n4nd0we'll be back after dinner19:52
blackburn1yeah I will be back later too19:52
CIA-52shogun: Sergey Lisitsyn master * r4520986 / src/shogun/lib/SGMatrix.h : Fixed SGMatrix constructor -
blackburn1okay seems almost no leaks at all now?20:42
-!- blackburn1 [~blackburn@] has quit [Quit: Leaving.]20:42
@sonney2kblackburn, blackburn2 or whoever - that kind of change is dangerous...20:47
@sonney2kin particular since SGVector etc all don't do it by default too (and I assume that this is used in the code)20:49
-!- blackburn2 is now known as gsomix20:54
gsomixsonney2k, good evening. how are you? :)20:57
@sonney2kgsomix, fighting with the webserver21:12
gsomixsonney2k, for new site?21:14
gsomixok, good luck with it :)21:27
-!- vaio [~vaio@] has joined #shogun22:27
vaioIs there anyone here?22:27
vaiohere ?22:27
vaioI am doing some exps using shogun-toolbox22:28
-!- vaio [~vaio@] has quit [Client Quit]22:29
n4nd0hello vaio22:32
n4nd0maybe next time...22:32
-!- n4nd0 [] has quit [Quit: leaving]23:03
-!- hoijui [] has quit [Quit: Leaving]23:15
CIA-52shogun: Sergey Lisitsyn master * r25ba47c / (2 files in 2 dirs): Another way to avoid memleak with dense features serialization -
--- Log closed Fri Aug 31 00:00:17 2012