Open in new window / Try shogun cloud
--- Log opened Mon Dec 05 00:00:59 2011
-!- blackburn [~blackburn@31.28.51.215] has quit [Quit: Leaving.]00:28
-!- shogun-buildbot [~shogun-bu@7nn.de] has quit [Quit: buildmaster reconfigured: bot disconnecting]02:39
--- Log opened Mon Dec 05 03:23:14 2011
-!- shogun-toolbox [~shogun@7nn.de] has joined #shogun03:23
-!- Irssi: #shogun: Total of 5 nicks [0 ops, 0 halfops, 0 voices, 5 normal]03:23
-!- Irssi: Join to #shogun was synced in 7 secs03:23
-!- in3xes_ [~in3xes@180.149.49.230] has joined #shogun04:14
-!- in3xes [~in3xes@180.149.49.230] has quit [Ping timeout: 244 seconds]04:18
-!- in3xes_ is now known as in3xes05:13
-!- in3xes [~in3xes@180.149.49.230] has quit [Ping timeout: 255 seconds]05:53
-!- blackburn [~blackburn@31.28.51.215] has joined #shogun07:48
-!- blackburn [~blackburn@31.28.51.215] has quit [Quit: Leaving.]08:13
-!- sonne|work [~sonnenbu@194.78.35.195] has joined #shogun11:40
-!- blackburn [5bdfb203@gateway/web/freenode/ip.91.223.178.3] has joined #shogun12:37
blackburnsonne|work: I have reproduced custom kernel error..13:06
blackburnsetting triangle from full fails, will try to fix today13:06
blackburnthen if you will manage to detect HMM or SVMs errors we can release13:06
sonne|workblackburn: hi14:05
blackburnhi14:05
sonne|workI noticed that 10 tests or so are failing on the train ride to work14:05
sonne|worknot the old tests but the newer ones (python tester.py )14:05
sonne|workI've debugged a few - but it still needs work also from you ...14:05
blackburnsonne|work: do you remember any?14:05
sonne|workall the converters have no test14:05
blackburnsonne|work: converters tests is difficult task14:06
sonne|workbasically for non-existent tests one needs to call python generator.py <filename_of_the_python_foo>14:06
blackburnsonne|work: I did tell you already, there is problems..14:06
sonne|workyeah but we need something reasonable... maybe you can come up with something14:06
blackburndamn, there are14:07
blackburnsonne|work: I can but I need time, I think it would be ok to do it later14:07
blackburnsonne|work: almost all of them requires to add some self-testing methods I can use in python14:07
sonne|workI think this is really important - otherwise we won't know if we broke anything14:07
blackburnsonne|work: I know status of converters 24h a week ;)14:08
sonne|workI don't have time now to discuss but lets do this maybe tonight14:08
blackburnsonne|work: but others - yes, we should fix it14:08
blackburnokay14:08
sonne|work(I thought so too about several other components btw)14:08
sonne|workcu14:08
blackburnI will be online from 23-30 UTC+414:09
blackburncu14:09
-!- blackburn [5bdfb203@gateway/web/freenode/ip.91.223.178.3] has quit [Quit: Page closed]15:19
-!- blackburn [~blackburn@188.168.4.192] has joined #shogun19:06
-!- blackburn [~blackburn@188.168.4.192] has quit [Ping timeout: 252 seconds]19:13
-!- blackburn [~blackburn@188.168.4.192] has joined #shogun19:14
blackburnsonney2k: here19:36
-!- shogun-buildbot [~shogun-bu@7nn.de] has quit [Quit: buildmaster reconfigured: bot disconnecting]20:12
-!- shogun-buildbot [~shogun-bu@7nn.de] has joined #shogun20:13
-!- mode/#shogun [+o sonney2k] by ChanServ20:17
blackburnsonney2k: I wonder why almost all tests are 'ERROR' :D20:17
blackburnbugs.. I haz it20:25
15SAAI18Mshogun: Soeren Sonnenburg master * r77c55d7 / (2 files): fix library and evaluation test - http://git.io/iCkWjA20:29
15SAAI18Mshogun: Soeren Sonnenburg master * r0a36522 / data : require new data for regression tests - http://git.io/Iy6JdA20:29
@sonney2kblackburn, here maybe 3-4 tests fail + all the converter_* ones20:29
blackburnsonney2k: 50% here20:29
blackburnjust like united russia election result20:29
@sonney2kunited we stand20:30
@sonney2kwait20:30
@sonney2kthis happens when you do python tester.py?20:30
blackburnsonney2k: ERROR OK ERROR OK ..20:30
@sonney2k?20:32
@sonney2kyes or no?20:32
blackburnsonney2k: ah sorry yes20:32
blackburnwait20:32
blackburnpython modular?20:32
@sonney2kyes the others don't support that20:33
blackburnthen yes20:33
@sonney2kI have an error only in preprocessor_dimensionreductionpreprocessor_modular.py + converter_*20:33
blackburnlol20:34
@sonney2kso which is the first test failing for you?20:34
blackburnregression_krr_modular.py setting 1/2                        ERROR20:34
blackburnregression_krr_modular.py setting 2/2                        ERROR20:34
blackburnregression_libsvr_modular.py setting 1/2                     ERROR20:34
blackburnregression_libsvr_modular.py setting 2/2                     ERROR20:34
blackburnregression_svrlight_modular.py setting 1/2                   ERROR20:34
blackburnregression_svrlight_modular.py setting 2/2                   ERROR20:34
blackburnthese ones for example20:34
@sonney2kyou don't have lapack/atlas?20:35
blackburnsonney2k: I thought I have20:35
@sonney2kor let me guess different cpu / optimization flags different results?20:35
@sonney2kI made it easy to investigate20:36
@sonney2kjust run python tester.py -d regression_krr_modular.py20:36
@sonney2kyou will get detailled debug output and be in pdb to investigate20:36
@sonney2kthere are two variables then a and b20:36
shogun-buildbotbuild #361 of r_static is complete: Failure [failed test_1]  Build details are at http://www.shogun-toolbox.org/buildbot/builders/r_static/builds/361  blamelist: sonne@debian.org20:36
@sonney2kboth should be lists20:36
@sonney2kwith the things returned by this function20:36
@sonney2kblackburn, btw did you do git submodule update?20:37
@sonney2knot that we have different versions of data...20:37
blackburnsonney2k: minute20:37
@sonney2kI now forcefully enabled all tests checks in buildbot20:38
blackburn[ -6.67244038e-14   9.76996262e-15   8.28848101e-12  -2.22044605e-1420:39
blackburn  -2.08366657e-12  -7.94919686e-14   4.06741307e-12  -5.29576383e-1420:39
blackburn  -1.99840144e-15  -4.55191440e-15   5.21804822e-15   1.78868031e-1220:39
blackburn   1.99840144e-15  -5.07416331e-12  -6.25055563e-14  -3.21964677e-1520:39
blackburn   2.42028619e-14   5.55111512e-16  -7.99360578e-15   4.71844785e-1620:39
blackburn   4.62185845e-13   6.66133815e-15   5.10702591e-15  -2.10942375e-1520:39
blackburn  -6.69064804e-12  -3.01980663e-14   0.00000000e+0020:39
blackburnmismatch20:39
blackburnhm not really much20:39
@sonney2kthat explains it though20:40
blackburnsonney2k: how?20:41
@sonney2kdifferent atlas etc lib / cpu etc and so different results20:41
@sonney2kthese things are strict 100% exact match tests to determine if anything changes...20:41
blackburnsonney2k: oh that's pretty bad way to test machine learning algos20:43
@sonney2kno20:44
blackburnsonney2k: I have no atlas btw20:44
@sonney2kit is very tough to figure out if your eps change is due to a bug or sth else20:44
@sonney2kso running things twice should give 100% same result20:44
blackburntwice sure, but different machines..20:45
@sonney2kof course this assumes you can init the random number generator etc20:45
@sonney2ksure20:45
@sonney2kthat's true but for that one could easily introduce a numerical comparison and allow for some slack20:45
blackburnsonney2k: I do not really know but I have x86 ubuntu here, while you have 6420:46
@sonney2ktester.py is not that difficult to read and whenever we have floats/numpy matrices we could do a lossy test (if enabled at cmdline)20:46
blackburnit does matter sometimes20:46
@sonney2kproblem is that it is not clear how much slack to allow20:47
@sonney2kfloating point crap is awful20:47
@sonney2ke.g. a 1e-16 difference might lead to a 1e-2 difference in e.g. svm training20:48
@sonney2kand this could easily be a newly introduced bug (like forgotten variable to add or so)20:48
blackburnsonney2k: I have added a little slack20:48
blackburndistances are ok now20:48
blackburnbut svms are not20:48
@sonney2koptional please20:49
@sonney2k(like the -d option with -s --slack)20:49
@sonney2kyeah20:49
blackburnkernels are halfly ok20:49
blackburnsonney2k: something is really wrong with subgradient, sgd and ocas20:50
@sonney2ksgd and ocas work just fine here20:50
@sonney2ksubgradient yes broken20:50
@sonney2kwell very random20:50
blackburnsonney2k: then everything should look ok20:52
blackburnsonney2k: have you ran ./test_all?20:52
@sonney2kyes - I'd like to get rid of these tests though20:52
@sonney2kthat is the stuff we wanted to get rid of for 1.0.020:53
@sonney2kbut failed20:53
blackburnsonney2k: how much errors did you have?20:53
@sonney2ksubgradientsvm, sgd (but this can be ignored we changed default args and that is tested), hmm, localalignment, oligo string, FK, TOP20:55
@sonney2kthat's it20:55
@sonney2kbtw we should have these new tests for python-static too20:55
@sonney2kbut hell no we dont' have them for matlab/octave/r/...20:56
blackburnsonney2k: I'm worried only about HMM :)20:57
blackburnI have never used to use it but20:57
@sonney2kblackburn, so can you come up with some way that we can test your dim red stuff?21:12
blackburnsonney2k: yes but not now21:12
@sonney2khow long do you think will you need? I would rather not want to release with new functions that dont' have tests21:12
blackburnsonney2k: I have to write complex tests21:12
@sonney2kwhy?21:12
shogun-buildbotbuild #63 of nightly_default is complete: Failure [failed test_1]  Build details are at http://www.shogun-toolbox.org/buildbot/builders/nightly_default/builds/6321:13
blackburnsonney2k: e.g. to test isomap I have to check if geodesic distances are ~21:13
@sonney2kcan't you just apply $DIMREDMETHOD and compute $SCORE and then compare?21:13
blackburnsonney2k: but for LLE I have to check something much more complex21:13
shogun-buildbotbuild #77 of nightly_all is complete: Failure [failed compile]  Build details are at http://www.shogun-toolbox.org/buildbot/builders/nightly_all/builds/7721:13
blackburnsonney2k: I'm really sure with LLE, etc21:14
@sonney2kso you want to compute sth like objective value?21:14
blackburnthree algos I'm not sure with are NPE, LPP and LLTSA21:14
blackburnsonney2k: yes21:14
blackburnsonney2k: my point is we should fix HMM and release21:16
blackburnsonney2k: do not much worry about dimreduction things, I do a lot of development and testing still21:16
blackburnand now everything looks ok21:17
blackburnbtw a little before it wasn't21:17
@sonney2kI am more worried about the dimred stuff than the hmm stuff :)21:22
blackburnsonney2k: ehmm why?21:24
blackburnsonney2k: I did tests on LLE, KLLE and both LTSA and KLTSA21:27
blackburnon different data21:27
blackburnand checked with scikits even21:27
blackburnyou underestimate too much21:27
@sonney2kno this is just experience - at some point any algorithm we have in shogun was correct and well tested but the bigger things get the harder it is to ensure that there are no side effects21:36
blackburnit is much faster, flexible and accurate that it was in shogun 1.021:38
@sonney2kso please do the test when you find the time - we can now but I am not too happy about it21:38
blackburnsonney2k: test what?21:38
blackburnsonney2k: do you want to hold release without tests?21:40
@sonney2kplease do the tests for converter_* / dimred*21:41
@sonney2kno, as I said we can release now21:41
blackburnsonney2k: but hmm??21:43
@sonney2kI will attempt to fix all tests except converter_* / dimred* first21:44
@sonney2kthen21:44
blackburnsonney2k: it can took more than 2 weeks21:44
@sonney2kcould you please commit the modifyied tester.py?21:46
blackburnsonney2k: ok21:46
blackburnsonney2k: should it support different tolerances?21:46
@sonney2kyes, just as arg21:46
blackburnsonney2k: btw numpy.all is bad pattern21:51
blackburnthere was near or so21:51
@sonney2kI only know any/all21:57
15SAAI18Mshogun: Sergey Lisitsyn master * r380f6e2 / testsuite/python_modular/tester.py : Added tolerance option for tester - http://git.io/UcRuRw22:06
@sonney2kblackburn, I would rather use the relative deviation e.g. (a-b)/(a+b) < eps instead of norm22:09
@sonney2kanyway time to sleep22:10
blackburnnorm of (a-b)/(a+b)?22:10
15SAAI18Mshogun: Sergey Lisitsyn master * r21324b9 / (3 files in 2 dirs): Faces application to dimreduction, added word embedding example - http://git.io/Hlu8CQ22:13
15SAAI18Mshogun: Sergey Lisitsyn master * r51970aa / testsuite/python_modular/tester.py : Changed tolerance to max - http://git.io/M_3ezQ22:20
15SAAI18Mshogun: Sergey Lisitsyn master * r8108a74 / examples/undocumented/python_modular/converter_multidimensionalscaling_modular.py : Added testing info for MDS into python example - http://git.io/F_8Fyw22:41
-!- blackburn [~blackburn@188.168.4.192] has quit [Quit: Leaving.]23:24
--- Log closed Tue Dec 06 00:00:10 2011