Open in new window / Try shogun cloud
--- Log opened Sat Jul 06 00:00:19 2013
-!- sanyam [uid10602@gateway/web/] has quit [Remote host closed the connection]00:43
-!- hushell [] has joined #shogun00:49
-!- nube [~rho@] has joined #shogun00:53
-!- iglesiasg [] has quit [Ping timeout: 276 seconds]01:10
-!- shogun-notifier- [] has quit [Quit: transmission timeout]01:21
-!- iglesiasg [] has joined #shogun01:31
-!- iglesiasg [] has quit [Client Quit]01:31
-!- iglesiasg [] has joined #shogun01:31
-!- mode/#shogun [+o iglesiasg] by ChanServ01:31
@iglesiasghave you guys seen this?
@iglesiasgI bet they didn't use our cover tree for the benchmark, but yes theirs ?_?01:43
-!- FSCV [~FSCV@] has joined #shogun02:03
-!- iglesiasg [] has quit [Quit: Ex-Chat]02:40
-!- pickle27 [] has joined #shogun02:46
-!- nube [~rho@] has quit [Quit: Leaving.]02:48
-!- nube [~rho@] has joined #shogun02:49
-!- van51 [] has quit [Quit: Leaving.]03:19
-!- FSCV [~FSCV@] has quit [Quit: Leaving]03:48
-!- nube [~rho@] has quit [Quit: Leaving.]03:54
-!- lambday [67157d36@gateway/web/freenode/ip.] has joined #shogun04:14
shogun-buildbotbuild #449 of nightly_default is complete: Failure [failed test]  Build details are at
lambdaypickle27: hi05:18
pickle27lambday: hey05:18
lambdaypickle27: how's it going?05:19
pickle27pretty good you?05:19
lambdaypickle27: pretty good too :D05:19
lambdaypickle27: hey could you please have a look at this -
pickle27lambday: yeah I just fixed that earlier this eve but it hasn't been merged yet05:20
lambdaypickle27: okay, thanks man :)05:20
pickle27yeah heres the latest05:21
lambdaypickle27: feels so good seeing travis green :D05:22
pickle27yeah haha05:22
pickle27lambday: I hate trying to debug travis when things run on my machines but Im getting better at it05:22
lambdaytell me about it !! :(05:22
pickle27that unit test was a weird fix05:22
lambdaybut that worked :D05:23
pickle27yeah I think it was a bad memcpy on my part from eigen3 back to an SGMatrix05:24
pickle27I don't know why my machines are so resilient to these bugs lol and Travis picks them up05:24
lambdaytravis is like an annoyed girlfriend.. gets mad at pretty much every single mistake :D05:25
lambdaybtw.. you were using eigen3 for matrix, and then copied the results back to SGMatrix?05:26
pickle27yeah thats what I had, turns out it was easier and cleaner to just leave it as eigen305:27
pickle27I usually create SGMatrix and then use eigen::map but had this one the other way around05:27
pickle27worked on my machine but I don't think it memcpy'd fully on travis which was the issue05:28
lambdayI see.. yes I too use map, seems easier :)05:28
pickle27yeah talking about it now Im really not sure why I had it that way in the first place05:29
lambdayhehe... been there just 2 days back! :D05:30
lambdaypickle27: will be back after breakfast.. for you its dinner time I guess05:31
pickle27haha its like almost bedtime lol05:31
pickle27lambday: catch ya later05:31
lambdaypickle27: hehe... ciao man!05:31
-!- pickle27 [] has quit [Quit: Leaving]06:11
-!- votjakovr [] has joined #shogun06:36
-!- sanyam [uid10602@gateway/web/] has joined #shogun06:52
-!- gsomix [~gsomix@] has joined #shogun07:30
gsomixsonney2k, good morning. finally I fixed my reader. preparing PR.07:32
gsomixI'm sorry for what I wasn't yesterday on the channel07:33
lambdaysonney2k: moin moin07:48
lambdaysonney2k: SGSparseMatrix is not under parameter framework, shall I add that?07:49
* gsomix away08:20
-!- nube [~rho@] has joined #shogun09:00
-!- shogun-notifier- [] has joined #shogun09:03
shogun-notifier-shogun: lambday :develop * 350616d / / (4 files):
shogun-notifier-shogun: rational approximation unit-test added09:03
shogun-notifier-shogun: Heiko Strathmann :develop * eddf5ca / / (4 files):
shogun-notifier-shogun: Merge pull request #1212 from lambday/feature/log_determinant09:03
shogun-notifier-shogun: rational approximation unit-test added09:03
shogun-buildbotbuild #1198 of bsd1 - libshogun is complete: Failure [failed test_1]  Build details are at  blamelist: lambday <>09:17
shogun-buildbotbuild #1199 of bsd1 - libshogun is complete: Failure [failed test_1]  Build details are at  blamelist: Heiko Strathmann <>09:29
shogun-buildbotbuild #1321 of deb3 - modular_interfaces is complete: Failure [failed test python_modular]  Build details are at  blamelist: lambday <>09:45
-!- van51 [] has joined #shogun09:58
-!- foulwall [~user@2001:da8:215:6100:acb5:df7a:527c:2a23] has joined #shogun10:17
shogun-buildbotbuild #1322 of deb3 - modular_interfaces is complete: Failure [failed test python_modular]  Build details are at  blamelist: Heiko Strathmann <>10:20
-!- foulwall` [~user@2001:da8:215:6100:acb5:df7a:527c:2a23] has joined #shogun10:33
-!- gsomix [~gsomix@] has quit [Remote host closed the connection]10:43
-!- foulwall` [~user@2001:da8:215:6100:acb5:df7a:527c:2a23] has quit [Quit: ERC Version 5.3 (IRC client for Emacs)]10:44
-!- foulwall [~user@2001:da8:215:6100:acb5:df7a:527c:2a23] has quit [Remote host closed the connection]10:44
-!- foulwall [~user@2001:da8:215:6100:acb5:df7a:527c:2a23] has joined #shogun10:44
-!- foulwall [~user@2001:da8:215:6100:acb5:df7a:527c:2a23] has quit [Ping timeout: 252 seconds]11:33
-!- lisitsyn [] has left #shogun []11:37
-!- shogun-notifier- [] has quit [Quit: transmission timeout]12:03
@sonney2klambday, that would be excellent!12:15
@sonney2klambday, and btw regarding implementing DenseFeatures for the complex type - so far everything I implemented in shogun got used sooner or later. In particular if it is data type related.12:16
@sonney2kSo it is not a time waste!12:17
van51sonney2k: hey12:22
van51sonney2k: should I make an option to normalize the vector if the user wants so and make a PR?12:23
van51sonney2k: and if so I'll normalize with just vector.size() which is an upper bound12:25
lambdaysonney2k: hey.. sorry I was away..12:29
lambdaysonney2k: alright I'll add support for complex64 in DenseFeatures... (and there were a few others)12:29
-!- lambday [67157d36@gateway/web/freenode/ip.] has quit []13:04
-!- votjakovr [] has quit [Remote host closed the connection]13:06
-!- votjakovr [] has joined #shogun13:07
-!- van51 [] has left #shogun ["PING 1373112359"]14:06
-!- foulwall [~user@2001:da8:215:6100:1422:e7f8:7364:2ed6] has joined #shogun14:26
-!- foulwall [~user@2001:da8:215:6100:1422:e7f8:7364:2ed6] has quit [Ping timeout: 245 seconds]14:43
-!- foulwall [~user@2001:da8:215:6100:895c:6f5:d9c:ece9] has joined #shogun14:56
-!- pickle27 [] has joined #shogun15:13
-!- van51 [] has joined #shogun15:25
-!- pickle27 [] has quit [Quit: Leaving]15:27
-!- foulwall [~user@2001:da8:215:6100:895c:6f5:d9c:ece9] has quit [Ping timeout: 264 seconds]16:05
-!- lambday [67157f36@gateway/web/freenode/ip.] has joined #shogun16:16
-!- shogun-notifier- [] has joined #shogun16:56
shogun-notifier-shogun: Kevin Hughes :develop * d40ed8e / / (5 files):
shogun-notifier-shogun: added FFDiag AJD algorithm and a unit test.16:56
shogun-notifier-shogun: Sergey Lisitsyn :develop * d38be81 / / (5 files):
shogun-notifier-shogun: Merge pull request #1211 from pickle27/develop16:56
shogun-notifier-shogun: added FFDiag AJD algorithm and a unit test.16:56
-!- FSCV [~FSCV@] has joined #shogun17:07
shogun-buildbotbuild #1200 of bsd1 - libshogun is complete: Failure [failed test_1]  Build details are at  blamelist: Kevin Hughes <kevin@kevin-M3920.(none)>17:12
-!- lisitsyn [] has joined #shogun17:14
shogun-buildbotbuild #1323 of deb3 - modular_interfaces is complete: Failure [failed compile python_modular]  Build details are at  blamelist: Kevin Hughes <kevin@kevin-M3920.(none)>17:15
-!- lambday [67157f36@gateway/web/freenode/ip.] has quit []17:24
shogun-buildbotbuild #1201 of bsd1 - libshogun is complete: Failure [failed test_1]  Build details are at  blamelist: Sergey Lisitsyn <>17:25
-!- nube [~rho@] has quit [Quit: Leaving.]17:31
shogun-buildbotbuild #1324 of deb3 - modular_interfaces is complete: Failure [failed test python_modular]  Build details are at  blamelist: Sergey Lisitsyn <>17:45
-!- foulwall [] has joined #shogun18:18
foulwallping sonney2k18:18
-!- van51 [] has quit [Read error: No route to host]18:29
-!- van51 [] has joined #shogun18:31
-!- foulwall [] has quit [Ping timeout: 268 seconds]18:41
-!- pickle27 [] has joined #shogun18:45
-!- pickle27 [] has quit [Client Quit]18:49
van51sonney2k: I19:40
van51sonney2k: I've started working on a streaming version of the HashedDocFeatures19:40
van51sonney2k: do these streaming have features *have* to use the CInputParser?19:41
van51sonney2k: because I don't see the current version of the Parser to be able to be used easily in text collections like the webspam19:42
van51I may be missing something though19:43
van51when you see this (and what I asked before) please comment and I'll check the logs later19:43
-!- van51 [] has left #shogun ["PART #gsoc-gr :PING 1373132672"]19:44
-!- shogun-notifier- [] has quit [Quit: transmission timeout]19:56
@sonney2kvan51, yes streamingfeatures is what you would have to support. if it doesn't work with webspam just yet then we just use a different dataset for now20:06
@sonney2kvan51, so just do the online streamingdoc*features trying to not have too much code duplication20:55
@sonney2kthe cygwin buildbot find g++ again21:05
@sonney2kno more compiler RIP21:05
lisitsynsonney2k: what's the reason?21:06
@sonney2koh I am upgrading cygwin21:06
@sonney2knew one has swig 2.0.921:06
@sonney2kand lua etc etc21:06
@sonney2kvotjakovr, any idea why fails on ?21:10
@sonney2kvotjakovr, could you add some debug code to let us figure out what is wrong?21:10
votjakovrsonney2k: hi, have the same problem with numpy 1.7.121:11
@sonney2ksonney2k, what do you mean? it is failing or what?21:14
votjakovrsonney2k: i mean, that i think that this problem depends on numpy21:15
lisitsynvotjakovr: really? why do you think so?21:15
@sonney2kvotjakovr, Checking for Python numpy support ... 1.6.221:15
@sonney2kvotjakovr, seems an old version is on the buildbot21:15
@sonney2kvotjakovr, IIRC you changed some things in GPs recently no?21:16
@sonney2kcouldn't that be the reason?21:16
@sonney2kI mean examples also needed to be changed so maybe it is just missing a test data regenerate21:16
votjakovrsonney2k: hmm... so why is diff empty?21:17
@sonney2kvotjakovr, no idea21:20
@sonney2kvotjakovr, sounds like bug in test framework21:20
@sonney2kvotjakovr, last working version was21:20
@sonney2kvotjakovr, git diff c868efbc9dc180433fce5b8bf04f77d56fc59c2a  shogun/regression/GaussianProcessRegression.*21:22
@sonney2kvotjakovr, maybe this helps ...21:22
votjakovrsonney2k: but i didn't change implementation of algorithms21:28
votjakovrsonney2k: so i think, that data should stay the same21:29
@sonney2kvotjakovr, who changed it then?21:29
votjakovrsonney2k: sorry, what is changed?21:30
@sonney2kvotjakovr, well please checkout c868efbc9dc180433fce5b8bf04f77d56fc59c2a and see if it still passes the test21:31
@sonney2kvotjakovr, if it does figure out what has changed21:31
@sonney2kit is bad to have a broken test ...21:31
@sonney2kso it is important to get it back to work21:31
-!- gsomix [~gsomix@] has joined #shogun21:32
gsomixgood evening21:32
@sonney2kgsomix, good evening! good to see you alive21:33
gsomixsonney2k, yeah. There was a storm in Moscow. However, I was saved by the castle of Catherine the Great. :)21:35
gsomixsonney2k, btw,
votjakovrsonney2k: i've rolled back to c868efbc9dc180433fce5b8bf04f77d56fc59c2a, then i've run tests, and this test is failed and diff is empty (i've reinstalled shogun) :(21:40
@sonney2kgsomix, did you just update?21:41
gsomixsonney2k, yep21:41
@sonney2kgsomix, looks pretty good now. lets see what travis says and I will merge21:45
@sonney2kvotjakovr, weird21:46
@sonney2kvotjakovr, only explanation would be some update of data or library then21:46
votjakovrsonney2k: i've figured, that on line 29 of return numpy.all(a == b) is false21:50
@sonney2kvotjakovr, has some option to -d IIRC to stop when sth differs21:51
@sonney2kyou can then use21:51
@sonney2kp a21:51
@sonney2kp b21:51
@sonney2kto see the two objects21:51
votjakovrhmm, yep, they are different22:01
@sonney2kok then now you 'only' have to figure out whats wrong22:02
votjakovrsonney2k: maximal difference is approx. 1e-1222:03
@sonney2kvotjakovr, if it is just a numerical issue then use some numpy.round trick!22:06
votjakovrsonney2k: Btw regenerating of the data helps :)22:11
@sonney2kif the 1e-12 is justified then we can use this as a fix22:20
shogun-buildbotbuild #996 of cyg1 - libshogun is complete: Failure [failed configure]  Build details are at  blamelist: lambday <>, Kevin Hughes <kevin@kevin-M3920.(none)>, foulwall <>, Soeren Sonnenburg <>, Thoralf Klein22:31
shogun-buildbot<>, Viktor Gal <>, Sergey Lisitsyn <>, Kevin <>, Heiko Strathmann <>, Fernando Iglesias <>, van51 <>, Roman Votyakov <>22:31
gsomixwow, how many22:31
votjakovrsonney2k: ok:) i'll try to find the reason...22:31
@sonney2kyeah I managed to get the buildbot back to fail22:32
@sonney2klooks like the update did help a lot :`-(22:32
gsomixsonney2k, green.
votjakovrwow! travis is green :)22:33
@sonney2kshogun-buildbot, force build 'cyg1 - libshogun'22:35
shogun-buildbotbuild #997 forced22:35
shogun-buildbotI'll give a shout when the build finishes22:35
@sonney2kgsomix, yeah travis doesn't do everything and no cygwin in particular22:35
-!- shogun-notifier- [] has joined #shogun22:40
shogun-notifier-shogun: Soeren Sonnenburg :develop * c392690 / examples/undocumented/python_modular/
shogun-notifier-shogun: don't fail bsd bot when json is not available22:40
shogun-notifier-shogun: Evgeniy Andreev :develop * 6e4b7db / / (13 files):
shogun-notifier-shogun: added CircularBuffer and LineReader22:41
shogun-notifier-shogun: Soeren Sonnenburg :develop * db2aecf / / (13 files):
shogun-notifier-shogun: Merge pull request #1130 from gsomix/feature/getline22:41
shogun-notifier-shogun: LineReader class for reading lines from file stream22:41
@sonney2kgsomix, finally!22:41
gsomixsonney2k, thank God. :)22:42
lisitsynthere is no such thing!22:42
@sonney2kgsomix, lisitsyn thank $DEITY then22:44
@sonney2kin my case DEITY=vapnik22:44
-!- lambday [67157c36@gateway/web/freenode/ip.] has joined #shogun22:44
lisitsynwell that's better than some unexistent thing ;)22:44
@sonney2klambday, what is your $DEITY? DEITY=HeikoS or vapnik?22:45
@sonney2klambday, and good evening!22:45
@sonney2klambday, you are free to choose22:46
@sonney2klambday, I mean choose me ;)22:46
lambdaysonney2k: :-o22:46
@sonney2klisitsyn, I manually rm -rf'd the build directory on cyg122:47
@sonney2klisitsyn, and now it builds22:47
* sonney2k sighs22:47
lisitsynsonney2k: haha22:47
@sonney2kthere might be some weird race condition22:47
lambdayexport DEITY="sonney2k" :-o22:48
lambdayecho $DEITY22:49
lambdaysonney2k: lisitsyn I have a problem :-22:49
lisitsynlambday: houston on the line22:49
gsomixin my case DEITY=random cool physicist or mathematician. And it seems I know not many cool computer scientists, hm.22:50
* gsomix z-z-z-z22:50
lambdaylisitsyn: huh?22:50
@sonney2klisitsyn, baikonur rather no?22:50
* lambday max(confused)22:50
shogun-buildbotbuild #997 of cyg1 - libshogun is complete: Success [build successful]  Build details are at
@sonney2klambday, or that
lisitsynlambday: what is the problem?22:51
lisitsynsonney2k: baikonur is kazakhstan actually you know ;)22:51
@sonney2klisitsyn, yes since borat22:52
lambdaysonney2k: whoa!! what was that? looks like our Indian festival "Diwali"22:52
@sonney2klambday, a notebook!22:52
shogun-buildbotbuild #1202 of bsd1 - libshogun is complete: Success [build successful]  Build details are at
@sonney2klambday, I thought that was your problem22:52
@sonney2kall hail shogun-buildbot !22:52
gsomixnite guys.22:52
lambdaysonney2k: HAHAHAHA =)22:52
@sonney2khe is good22:52
@sonney2kgsomix, nite22:52
lambdayluckily, I use insti's computer for shogun stuff... my laptop is safe so far :D22:53
@sonney2klisitsyn, this one Borat: Cultural Learnings of America for Make Benefit Glorious Nation of Kazakhstan (2006)22:53
@sonney2klambday, heh22:53
lambdaysonney2k: lisitsyn: the problem is -22:53
lisitsynsonney2k: is that the full name of the movie?22:53
lambdayisn't that the movie maker or something22:53
lambdayhe's hilarious22:53
lambdaywell, the problem is :(22:54
lisitsynthat's a role of sacha baron-cohen  lambday22:54
@sonney2klisitsyn, yes22:54
lisitsynjust like admiral general aladeen :D22:54
lisitsynI don't know what is more hilarious22:54
@sonney2klambday, just tell us22:55
lambdaylisitsyn: I only know "that's" "a" "role" "of" "lambday" of what you just said :D22:55
lambdaysonney2k: I need to work with sparse matrices22:55
@sonney2kwe are drunk and entropy increasing at this time of day anyways22:55
@sonney2klambday, do it!22:55
@sonney2kso what is the problem22:55
@sonney2kI do that to22:55
@sonney2ktoo tuuut22:55
lambdaysonney2k: the problem is... :D22:55
lambdaysonney2k: I need to convert it to eigen3 sparse each time I use that22:56
* sonney2k grabs a beer and prepares for the worst22:56
lisitsynwe had a problem with performance and we went for multithreading22:56
@sonney2klambday, uhh ok big one22:56
lambdaysonney2k: you better.. I'm gonna drop a bomb22:56
lisitsynnwo ew h vea a wto bplemros22:56
@sonney2klisitsyn, any experience with that so far22:57
lisitsynsonney2k: sparse?22:57
-!- gsomix [~gsomix@] has quit [Ping timeout: 268 seconds]22:57
@sonney2ksparse from eigen3 I mean22:57
@sonney2kwhat format do they internally use22:57
lambdayso, the number of times I have to convert that is HUGE.. and each time new memory will be allocated and will be real bad performance wise22:57
lambdayand here is the bomb22:57
lisitsynCCR sonney2k22:57
@sonney2klambday, yeah not possible that is clear22:57
lambdayI want to use eigen22:58
lambday3s' sparsematrix as a member22:58
lambdayand want serialization/param registration :(22:58
lambdaywith that22:58
* lisitsyn kills himself22:58
lambdaylisitsyn: no don't die :(22:58
lambdayme cries22:58
* lambday cries22:59
lisitsynlambday: hah22:59
@sonney2klets all wine like real men!22:59
lisitsynwe should go for SURGERY22:59
* sonney2k grabs another beer22:59
lambdayshit man! I am missing all the drinks.. :(23:00
lambdayI don't have any atm :'(23:00
lisitsynlambday: I am into the eigen3 code now23:00
lambdaylisitsyn: which code?23:00
@sonney2klambday, how big is that matrix?23:01
lisitsynlambday: sparse23:01
@sonney2klambday, there are a couple of options23:01
lambdaysonney2k: say, a mil x a mil23:01
@sonney2ka) not use eigen3 for sparse23:01
lambdaysonney2k: what else we're gonna use?23:01
lambdayI need to run iterative solvers with that :(23:01
@sonney2kb) use eigen3 for sparse but convert it *once* from SGSparseMatrix23:02
lambdayI like the second..23:02
lambdaybut then I need to keep it as a param23:02
lisitsynlambday: all you need is love^W23:02
@sonney2kc) not use SGSparseMatrix except for serialization23:02
@sonney2klambday, no I mean how big is the matrix in terms of non-zero elements23:02
lisitsynCompressedStorage::Map(Index* indices, Scalar* values, size_t size)23:02
@sonney2klambday, mil x mil is nothing for a really sparse matrix :)23:02
lambdaysonney2k: wait let me check23:03
@sonney2klisitsyn, shogun uses LIL :)
lambday200,000 nz elements23:03
lisitsynsonney2k: I know23:04
@sonney2klambday, so small23:04
lisitsynsonney2k: I believe its conversion is fast23:04
lisitsynnobody would ever notice that23:04
lambdaysonney2k: but that's the demo that we're gonna prepare23:04
@sonney2klambday, in this case conversion is very cheap23:04
lisitsynI construct sparse matrices in tapkee dynamically23:04
@sonney2klambday, do you know how big matrices might become23:04
lisitsynit is like less than 0.01s or so23:05
lambdayHeiko knows23:05
@sonney2klisitsyn, yeah even less I think23:05
lisitsynlambday: serious things happen in solvers I guess rgiht?23:05
@sonney2klambday, conversion is only a problem when you have hard memory constraints23:05
@sonney2klambday, but this here is just 1MB23:05
lambdaysonney2k: wait, that can't be right! the matrix I have for ozone is about 32 mb23:06
lisitsynlambday: what I suggest now23:06
lisitsynuse construct from triplets thingy23:06
lisitsynif it is slow (measure first!)23:06
@sonney2klisitsyn, does eigen3 support using CRS / CCS based matrices23:06
lisitsynsonney2k: if you have arrays of indices and values23:06
lambdaysonney2k: lisitsyn and the main problem is - I need to do this convert this like 1,000,000x20 times23:06
@sonney2klambday, lisitsyn, if so we should have a converter from LIL -> CRS / CCS23:06
lisitsynlambday: why?23:06
lambdaylisitsyn: sonney2k I already added an util to convert to eigen3 sparse in eigen3.h23:07
@sonney2klambday, yeah why?23:07
lambdaysonney2k: I can convert it once and store inside the class as a member.. but that requires eigen3 sparse to be make it as a member23:07
lambdayno serialization no cloning nothing is possible23:08
lisitsyn20M times?23:08
lambdaywe'll be needing that I guess for parallel computation thing - same framework shoudl work23:08
lambdaysonney2k: lisitsyn we actually need to compute many many estimates of log-det23:08
lambday20 for each23:08
lambday1 mil is the number of estimates I am guessing23:09
lisitsynwhat 1M comes from?23:09
lambdayoh no23:09
lambdayI was wrong!23:09
lambdayit should be 1,000,000x20x10023:09
lisitsynyou need to solve23:10
lisitsyn1M x 1M sparse matric23:10
lisitsyn1M times23:10
@sonney2klisitsyn, don't forget it is /really/ sparse23:10
lisitsynsonney2k: no I understand but it is like a lot of time anyway23:10
@sonney2klambday, I still don't understand how exactly this is working23:11
lambdayI don't need to solve, but I need to convert it to eigen3-sparse from SGSparse this number of time23:11
lambdaysonney2k: well23:11
lisitsynlambday: okay why to convert it so many times?23:11
lambdayinitial plan was to keep SGSparseMatrix as a member of our opertor class23:11
lisitsynahh so you apply operator23:11
lisitsynand when you apply operator23:11
lisitsynyou convert23:11
@sonney2klambday, ahh ok23:11
lambdayeach time I need to apply, I need to convert23:11
@sonney2klambday, well a few more ideas23:12
lisitsynyes I have too23:12
lambdayand if I don't get to keep eigen3's sparse inside the class, we're stuck23:12
lisitsynsonney2k: you first23:12
lisitsynlambday: there is a way to put it inside23:12
lambdaysonney2k: please :(23:12
@sonney2ka) you could call finalize() or sth to make the conversion back to SGSparseMatrix23:12
shogun-buildbotbuild #1325 of deb3 - modular_interfaces is complete: Failure [failed test python_modular]  Build details are at  blamelist: Soeren Sonnenburg <>23:12
@sonney2kb) we could add a different storage layout to SGSparseMatrix23:12
@sonney2k(as in CCS, CRS)23:13
@sonney2kif eigen3 can immediately use this23:13
lambdaythat would be great!23:13
lambdaylike map for dense23:13
lambdayours is an adjacency list, right23:13
lisitsynmy other solution is23:14
@sonney2klisitsyn, your ideas?23:14
lisitsynjust add a base class in operator23:14
lisitsynlike MatrixHandler23:14
lisitsynand declare EigenMatrixHandler in .cpp23:14
lisitsynand use it inside23:14
@sonney2klisitsyn, so it would be making eigen3 work with LIL?23:15
lambdaylisitsyn: I didn't get.. I need to store it as a member.. AND also probably need serialization23:15
lisitsynsonney2k: I mean it would convert once23:15
lisitsynlambday: ahh serialization sorry23:15
lambdaysonney2k: lisitsyn can't we add Eigen::SparseMatrix as a ctype or something?23:15
@sonney2ktaht would be option c) then - make eigen3 support LIL23:16
lambdayLIL=? :(23:16
lisitsynlist of lists23:16
@sonney2klambday, our sparse matrix format23:16
lambdaybut when we need to multiply vectors with it, I think LIL is not a good option23:16
@sonney2klambday, why not?23:17
lambdayI mean, if I can directly use eigen3's sparse, I don't need SGSparseMatrix at all for our purpose23:17
lisitsynlambday: what do you use eigen3 here ?23:18
@sonney2klambday, yeah problem is with serializtion only23:18
lambdaymat-vec multiplications23:18
lambdaywhich is really fast23:18
lisitsynjust multiply?23:18
lambdayfor iterative solvers, yes23:18
lisitsynthere is no special tricks here23:19
lambdayand that's what we'll be using sparse for, so yes23:19
lisitsynlambday: are you computing largest eigenvalues?23:19
@sonney2klambday, ok in this case we have d) do multply with shogun's code23:19
lambdaylisitsyn: yes that, and also writing a cg solver23:19
lisitsynlambday: are you going to estimate smallest too?23:19
lambdaysonney2k:you mean, add  a new method with SGSparse?23:20
lambdaylisitsyn: yes, extremal23:20
lambdaybut that's not where I'll be using it only..23:21
@sonney2klambday, it has it already IIRC23:21
* sonney2k checks23:21
lisitsynlambday: then you'd need to solve systems too?23:21
lambdaylisitsyn: yes that's the main part23:21
lambdayeach estimate generates like a lot of systems which need to be solved using iterative solvers23:21
lambdayfor sparse23:21
lisitsynare you writing your own solver for that?23:21
@sonney2klambday, yes you could do sparse_matrix[index].dense_dot()23:21
lambdayyep, have to23:21
lambday(well, taking krylstat's)23:21
shogun-buildbotbuild #1203 of bsd1 - libshogun is complete: Failure [failed test]  Build details are at  blamelist: Evgeniy Andreev <>, Soeren Sonnenburg <>23:21
lisitsynlambday: why have to btw?23:22
lambdaysonney2k: that computes mat-vec product?23:22
lambdaylisitsyn: we need a solver that solves a family of shifted systems simultaneously23:22
lambdayfor complex23:22
lisitsynI see23:23
lambdayI don't think eigen3 provides that23:23
lisitsynhaven't seen such thing there23:23
lambdaybut mat-vec product is the key here and we gotta do it many many times23:23
@sonney2klambday, well just for one row.23:23
lambdayI want it to be fast :(23:23
* lambday scratches his head23:23
@sonney2klambday, but you could easily add a function to SGSparseMatrix using that then23:24
lisitsynmost of things that make code fast are not that obvious and you'd have to measure anyway23:24
@sonney2kit is easy and will be fast23:24
@sonney2kthe dense_dot has jus non-zero elements in vector operations23:24
lambdaysonney2k: we'll be needing sparse-matrix-dense vec products23:25
lambdayand also sparse-matrix-sparse-vector products23:25
lisitsynbah boeing 777 crashed at SFO23:25
@sonney2klambday, we have that23:26
@sonney2klambday, sparse_dot()23:26
lisitsyn*just* two dead23:26
lambdaysonney2k: alright I'll check then23:26
lambdayso, not use eigen3 at all??23:26
@sonney2klisitsyn, that is where we will arrive for GSoC summit23:27
lambdaywhen is it?23:27
lisitsynsonney2k: I am on the waiting list :P23:27
@sonney2klisitsyn, lucky you23:27
@sonney2klambday, it should take you no more than a few minutes to add these 2 funcitons to SGSparseMatrix23:27
lambdaysonney2k: lisitsyn oh but the problem is.. we don't have serialization with SGSparseMatrix either23:28
@sonney2klambday, just use dense_dot and sparse_dot from SGVector for that23:28
lambdayis there any reason it wasn't added?23:28
lisitsynlambday: oh well it is easier though23:28
lambdaysonney2k: alright then23:28
@sonney2klambday, then you can measure how fast things are for eigen3 / sgsparse* for your application23:28
lisitsynlambday: the only reason is not enough hours in day of sonney2k and lazyness of me23:28
lambdaysonney2k: okay23:29
@sonney2kno idea if eigen3 is doing it faster currenlty23:29
lambdaysonney2k: I think they do :-/23:29
lisitsynsonney2k: what exactly?23:29
lambdaymat-vec products, right?23:29
@sonney2klambday, problem with sparse matrices is that you don't have contiguous memory access23:29
@sonney2kso I don't expect taht23:29
lisitsynI think their storage is more cache-efficient23:30
lisitsynit is contiguous23:30
@sonney2klisitsyn, ours is contiguous23:30
@sonney2kthey have to look into different index tables all the time23:30
lisitsynsonney2k: but our columns are not continuous right?23:30
@sonney2kbut we have index / value contiguos in memory23:31
lisitsynof a column23:31
lisitsynnot of the whole matrix, right?23:31
-!- lambday_ [67157f4c@gateway/web/freenode/ip.] has joined #shogun23:31
lambday_sonney2k: lisitsyn sorry I got dc23:31
lisitsyneigen's storage is just two arrays23:31
@sonney2klisitsyn, well malloc will usually allocate it contigously23:31
lisitsynwe have an arrays of contiguous arrays23:31
@sonney2kyes true23:32
lisitsynsonney2k: still a room for cache misses23:32
@sonney2klisitsyn, for a nnz=100k matrix theirs might be in cvache completely23:32
@sonney2klisitsyn, I don't expect CCS / CSR to be better to the contrary23:32
lisitsynI have no idea what is better23:33
@sonney2klisitsyn, I mean if we allocated one block of memory23:33
@sonney2kand then had the lists in there contigously23:33
@sonney2kthen we would certainly win23:33
@sonney2kand I guess malloc does just that normally23:33
-!- lambday [67157c36@gateway/web/freenode/ip.] has quit [Ping timeout: 250 seconds]23:34
lisitsynI don't know if compilers optimize that23:34
lisitsynbut you still kind of jump here23:34
lambday_is there any way we can test that?23:34
lisitsynfrom array of pointers to the array of values23:34
lisitsynlambday_: hmm you can indeed23:35
lisitsynlambday_: multiply some matrix as SGSparsematrix and as Eigen3 sparsematrix23:35
lisitsynwith some vector or so23:35
lisitsynand check cache misses with valgrind ;)23:35
@sonney2klisitsyn, look at this23:36
lisitsynand walltime and other things like that23:36
lambday_I was trying to find some page that I saw probably, regarding mat-vec products23:36
@sonney2kint main(int argc, char** argv)23:36
@sonney2k    void* a=malloc(256);23:36
@sonney2k    void* b=malloc(256);23:36
@sonney2k    printf("%p %p\n", a,b);23:36
lambday_not sure where did I see that23:36
@sonney2k0x1dcb010 0x1dcb12023:36
lisitsynsonney2k: yes I understand that23:36
@sonney2kbut there might be other tricks23:37
@sonney2kI think cache misses might rather come from accessing the million dim vector23:37
lisitsynI am just afraid it reloads the cache23:37
@sonney2kit is randomly jumping in there23:38
lisitsynwhen working with list of lists and the list23:38
@sonney2klambday_, ok we are guessing too much now. just try it out23:38
lambday_but the entries are SparseVectorEntries, can be created randomly, not col by col.. and then added to particular row or col.. so when we try to access it sequentially, it can be slow23:38
@sonney2klisitsyn, should we have some directory in shogun/benchmarks?23:39
@sonney2kor so where we do such experiments?23:39
lisitsynsonney2k: not in code right?23:39
lisitsynin examples I guess23:39
@sonney2klisitsyn, no23:39
lisitsynthey are typically not in code I think23:39
@sonney2kas in shogun/tests shogun/examples shogun/benchmarks23:39
lisitsyndidn't get you23:39
@sonney2klambday_, so please add it there23:40
@sonney2klambday_, van51 can add his dotfeatures benchmarks there too23:40
lambday_sonney2k: umm... what will go there in the benchmarks?23:41
lambday_I mean, I shouldn't add the dot in lib/SGSparseMatrix.h ?23:41
@sonney2klambday_, the mil x mil SGSparsematrix with nnz=100k times dense vector run say 100 times23:41
lambday_or just the experiment code goes there23:41
@sonney2klambday_, yes23:41
@sonney2kto both23:41
@sonney2klambday_, so feel free to do the same test with eigen3 there and compare23:42
@sonney2kI am curious what the result will be23:42
@sonney2klambday_, so just do it for the most needed operation for now (and only implement that)23:43
@sonney2kso when we figure out it doesn't work we can do it differently again23:43
lambday_sonney2k: alright then23:43
lambday_in the krylstat code (that erlend wrote) they use eigen3 everywhere.. ours shouldn't be slower than that23:44
lambday_I'm sorry again. our mil x mil sparse matrix would have at least a mil nz entries (main diagonal)23:46
lambday_anyway, I'm trying23:47
@sonney2klambday_, try to create a 'realistic' sparse matrix ohh and note that sparse vector indices must be sorted23:50
lambday_sonney2k: yes23:51
shogun-buildbotbuild #1326 of deb3 - modular_interfaces is complete: Failure [failed test python_modular]  Build details are at  blamelist: Evgeniy Andreev <>, Soeren Sonnenburg <>23:59
--- Log closed Sun Jul 07 00:00:20 2013