Open in new window / Try shogun cloud
--- Log opened Mon Jan 20 00:00:00 2014
-!- shogun-notifier- [] has quit [Quit: transmission timeout]00:59
-!- sonne|osx_ [] has joined #shogun03:29
-!- sonne|osx [] has quit [Ping timeout: 272 seconds]03:31
-!- sonne|osx_ is now known as sonne|osx03:31
shogun-buildbotbuild #686 of nightly_default is complete: Failure [failed test]  Build details are at
-!- shallhome [~shallhome@] has joined #shogun04:40
-!- shallhome [~shallhome@] has quit [Remote host closed the connection]05:14
-!- sonne|osx [] has quit [Quit: sonne|osx]07:21
-!- sonne|osx [~sonne@] has joined #shogun07:49
sonne|osxbesser82: we have the same pickle problem again07:54
sonne|osxbesser82: please enlighten me how you fixed it!07:54
-!- lisitsyn [] has joined #shogun07:56
-!- sonne|osx [~sonne@] has quit [Ping timeout: 252 seconds]08:00
-!- sonne|osx [~sonne@] has joined #shogun08:04
-!- sonne|osx [~sonne@] has quit [Client Quit]08:05
-!- sonne|osx [] has joined #shogun08:19
-!- sonne|osx [] has quit [Client Quit]08:21
besser82sonne|work:  which pickle-problem @ which test / build?  I can haz linxx, plz?09:43
sonne|workbesser82: well all python3 tests fail due to me adding  the pickle string based comparison09:46
sonne|workbesser82: so IIRC you figured there is some encoding problem....09:47
besser82sonne|work:  encoding ain't the only one  ;)09:47
besser82sonne|work:  but in fact the major one09:47
besser82sonne|work:  lemme cycle through all recent commits to see what changed and I'll give you some advice  ;)09:48
sonne|workbesser82: well it is just this pickle.dumps(objs) == comparison in tester.py09:50
besser82sonne|work:  then there is no general solution (tm);  the most problematic thing is the incompat between numpy and py3-numpy, which can't be solved without external workaround like storing the numpy-datatypes with numpy's internal serialization methods, unfortunately.  :(10:41
sonne|workbesser82: well all I want is that this doesn't crash!10:41
besser82sonne|work:  you mean the segfaults, don't you?10:42
besser82sonne|work:  those were never fixed IIRC10:42
-!- lambday [67157f36@gateway/web/freenode/ip.] has joined #shogun10:43
sonne|workbesser82: I thought you did sth with the encoding?10:43
besser82sonne|work:  because of we cannot restore py2_mod stuff with py3_mod...10:43
besser82sonne|work:  I did some experiments with that, but that actually wasn't successful  :(10:44
besser82sonne|work:  It's not only the encoding / datatypes used in pickle; it's basically more  :(10:44
besser82sonne|work:  I tried sth. with manually hex-editing the pickles to have str-data stored as py3-bytes, but then there is some other serious trouble, when opening them with py3_modular  :(10:46
besser82sonne|work: the only way I see to get that fixed is to have different pickles for py2 and py310:48
besser82sonne|work: but hoding the same data...10:48
besser82sonne|work:  which brings us to the next question:  How to get that done?10:49
besser82sonne|work:  I mean, those pickled must have been created somewhen; but how?10:49
sonne|workbesser82: well it is just some ascii file that we create11:07
sonne|workbesser82: how would embed some ascii string python2/3 compatible11:07
adrinsonne|work, besser82: hey, what's up? Good morning.11:24
sonne|workadrin: good morning11:24
besser82sonne|work:  basically I would try to use some py3::bytes-type which is perfectly binary-compat with char*11:24
sonne|workadrin: we are trying to get python3 to work ...11:24
besser82sonne|work:  but the problem here is py2 doesn't know "bytes" and has str only, which is the same as char*, too11:25
adrinif you put a "b" before the string, it will be interpreted as a byte-char in py3, and will be ignored in py2.7, why doesn't that work here?11:25
sonne|workadrin: who says it wouldn't :)11:26
besser82adrin:  Hey! G'Mornin!  Yes, as sonne|work says all about Python3 && shogun11:26
sonne|workadrin: we are just not python3 experts11:26
sonne|workthat is something we can fix at tonights stammtisch :)11:26
adrinbesser82, sonne|work: there are two prefixes for strings in python, "u" and "b"11:26
sonne|workadrin: in python 2.6 etc too?11:26
adrin"u" is ignored by py3, and "b" is ignored by 2.711:27
adrinno py2.6 doesn't know about them11:27
adrinsonne|work: these two were introduced in py2.7, to make people able to have strings compatible in both 2 and 3 pythons, but you will need to have at least py2.711:27
besser82sonne|work, adrin:  for Py2.6 one needs to `import __future__` within some exception11:28
besser82sonne|work, adrin:  like try: import __future__ except: #noop11:28
sonne|workadrin: how do we get these if we are reading a file?11:28
besser82sonne|work, adrin:  but that won't fix the problem with pickles  :-P11:29
adrinsonne|work: so the problem is that you want to read a file and you need to make sure you're interpreting it as ASCII bytes?11:29
besser82adrin: no, we are importing a pickle made with proto 211:30
besser82adrin: and for some reason py3_modular segfaults when restoring the py2_mod data from that pickle  ;)11:30
sonne|worksry no time for that jsut now11:30
sonne|workadrin: will you be around in the evening?11:30
adrinsonne|work: yeah sure. I'll try to see what I can do until then.11:31
adrinbesser82: In the mean time, it would be nice if you could send me a py2_mod data so that I can try to load it.11:38
besser82adrin:  you can fetch them from here:
besser82adrin:  here is a list of those failing / segfaulting tests:
adrinbesser82: thanks11:40
besser82adrin:  the BIG problems is the binary-incompat of numpy-arrays && those segfaults, when restoring data from python_modular of shogun  ;)  np.  You're welcome  :D11:40
-!- lambday [67157f36@gateway/web/freenode/ip.] has quit [Ping timeout: 272 seconds]11:41
adrinbesser82: I see11:41
besser82adrin: btw. manually converting str to bytes in pickle didn't work.  :(11:42
besser82adrin: like sed -e 's!^S\'!B\'!g' pickle.file11:43
-!- bb_ [458fc8de@gateway/web/freenode/ip.] has joined #shogun15:19
-!- bb_ [458fc8de@gateway/web/freenode/ip.] has quit [Ping timeout: 272 seconds]15:39
-!- sonne|osx [~sonne@] has joined #shogun15:40
sonne|osxadrin, besser82 btw we can of course generate the data again with python2.7 or python3 if that helps15:46
besser82sonne|osx:  that would be the easiest solution  ;)15:46
besser82sonne|osx:  we need separate pickles for py2 && py3 and all would be fine15:47
sonne|osxbesser82: would this ensure that it can be read with python2.7 then?15:47
sonne|osxbesser82: nah15:47
sonne|osxone for all15:47
besser82sonne|osx:  one for all isn't possible ---> numpy15:47
sonne|osxbesser82: so there is no way to load numpy data from python3?15:47
besser82sonne|osx:  it is when pickle is from py315:48
besser82sonne|osx:  they are not interchangeable15:48
besser82sonne|osx:  that's why numpy has own im/export methods for data  ;)15:48
sonne|osxbesser82: and stuff like pickle.load(file_obj, encoding='latin1') ?15:48
besser82sonne|osx:  will be readble, but numpy data is a mess then15:49
besser82sonne|osx:  as to be seen in tesstsuite on py3, which fail but don't segfault  ;)15:49
sonne|osxbesser82: hmmhh btw will you be around for the stammtisch tonight?15:49
besser82sonne|osx:  sure!15:49
besser82sonne|osx:  not till the end, but at least ~2 h15:50
sonne|osxyeah sure15:51
sonne|osxbesser82: no idea why the numpy people did it this way15:52
sonne|osxsucks badly that you cannot load your old data15:52
besser82sonne|osx:  you can but only with some extra steps15:52
sonne|osxbesser82: which are?15:52
besser82sonne|osx:  you need to load from the py-ver the pickle was made with and then export the numpy-data with the export-method of numpy15:53
besser82sonne|osx:  that export is gurateed to be readable from every recent python15:53
sonne|osxbesser82: which would be savenpy? or what15:53
besser82sonne|osx:  load pickle15:55
besser82sonne|osx:  $$file_handle)15:55
besser82sonne|osx:  open other python-ver15:56
besser82sonne|osx:  create empty numpy-array15:56
besser82sonne|osx:  $new_array.load($my_export_from_old_pyver)15:56
sonne|osxbesser82: not good - maybe we can work around this with python's stringIO15:57
sonne|osxadrin: what do you think? Maybe we store numpy stuff with stringIO - then we could keep one single format?15:58
besser82sonne|osx: why using stringIO?15:58
sonne|osxbesser82: string that behaves like a file handle15:58
besser82sonne|osx: i mean numpy has own methods for that... ???15:59
sonne|osxbesser82: so we could write that string out instead of pickle15:59
sonne|osxand it would be compatible15:59
besser82sonne|osx: that can be done with those numpy.methods, too15:59
besser82sonne|osx: those are gurateed to be compat within different python-interp16:00
sonne|osxbesser82: lets talk about that in the evening16:00
besser82sonne|osx:  kk16:00
* sonne|osx off train16:00
-!- sonne|osx [~sonne@] has quit [Quit: sonne|osx]16:00
-!- lambday [67157e37@gateway/web/freenode/ip.] has joined #shogun16:04
-!- parijat [671b082a@gateway/web/freenode/ip.] has joined #shogun16:10
adrinbesser82: me and my office mate were looking at the errors/data16:11
besser82adrin:  Many thx!  and what did you find out?16:12
adrinbesser82: he's never had such a problem with pickle, but he never used different python versions16:12
besser82adrin:  same for us, too  ;)16:12
adrinbesser82: nothing yet, but we're trying to see how to tell pickle how to interpret strings16:13
besser82adrin:  thanks for your work  ;)16:13
adrinbesser82: I personally prefer to handle it that way, or generating the data using utf strings, and tell pickle in py2 to interpret them as utf, I guess then there is no need for multiple data files, right?16:14
besser82adrin:  would that work with numpy-arrays?16:14
adrinbesser82: don't know yet, but we have py2.6, 2.7, and 3.3 available. Personally I don't recommend to worry/support 2.6. 2.7 and 3.3 should be enough16:16
besser82adrin:  I would apprechiate support for py2.6 because of RHEL6  ;)16:16
besser82adrin:  But those details i might add somewhen later then  ;)16:16
adrinbesser82: I see. Then we'll test 2.6 too. Shall I ask for a peice of code which generates the pickle data + a sample saved data + the code which is supposed to read it? We found many files which can't read a data, but it would be nice if we had just an example, if available. If not, we'll try with a simple string/np.ndarray/set/list file if even with such a simple example the problem perists.16:19
besser82adrin:  I think sonney2k has that code or some examples about generation....16:21
besser82adrin:  but trying with some toy data would at least point into the right direction, I think.16:21
adrinbesser82: ok, we'll do it. BTW, is the stammtisch a real physical one or you guys were talking about a virtual stammtisch? :D16:23
-!- gsomix [~gsomix@] has quit [Ping timeout: 264 seconds]16:28
besser82adrin:  virtual one on irc  ;)16:33
besser82adrin:  distances are a bit problematic  ;)16:33
besser82adrin:  you, toralf and sonney2k possibly could meet physically Berlin  ;)16:34
besser82adrin:  me is from Hildesheim (30 km south of HAnnover)16:34
-!- snik [] has joined #shogun16:37
besser82snik:  hi!  You must be one of adrin's office mates, right?!?16:37
snikHi! besser82, yes, i am!16:38
besser82snik:  nice to meet you!  :D16:39
snikNice to meet you too! :)16:39
besser82snik:  and thank for helping with that py3-thing!  :D16:39
adrinbesser82: haha, we're based in Saarbruecken, Berlin is a bit far :D16:39
snikoh, we have just started..16:39
besser82adrin:  kk, that's a bit far to Berlin, too  ;)16:40
adrinbesser82: why the 'latin_1' decoding option for numpy arrays is not an option here? I didn't understand why it's not desireable.17:53
besser82adrin:  the problem is: numpy-array-pickles are not binary compatible between py2 and py3  :(17:54
besser82adrin:  one can encode / decode the pickled-data using 'latin-1', but the outcome is just garbage17:55
adrinbesser82: ok, let me see, it didn't seem garbage, let me check17:56
adrinbesser82: my mistake, it IS garbage. will be back soon :D17:57
snikok, right, it really is!17:57
besser82adrin, snik: works between pyvers, but pickle seems not.  :(17:58
-!- genrut [~genrut@] has joined #shogun17:59
snikbesser82: pickle is really an issue, visibly even for the larger community, especially with the protocol thingy18:00
besser82snik:  unfortunately, yes  :(18:00
-!- genrut [~genrut@] has quit [Ping timeout: 246 seconds]18:09
-!- parijat [671b082a@gateway/web/freenode/ip.] has quit [Ping timeout: 272 seconds]18:40
adrinbesser82: I guess we failed :D, the problem is the ndarray representation is in unicode format, and can not be read in python 218:56
adrinbesser82: we tried to save using py3, and load in py2, but for this reason it doesn't work18:57
besser82adrin:  save like pickle?18:57
besser82adrin:  or
adrinbesser82: yeah, in theory it should work if the protocol is 2, but it doesn't18:57
adrinbesser82: did you guys try the I would vote for them if they work between python versions18:58
adrinbesser82: we thought if we could save the data in py3 in a way that py2 could read it, then somehow the problem is solved, but even this solution didn't work18:59
besser82adrin:  let me discuss that with sonney2k on tonight's stammtisch...18:59
besser82adrin: the other ugly thing is those segfaults of python3_modular, when trying to recreate data from py2 pickle...  :(19:00
besser82adrin:  any thoughts on that?19:00
snikbesser82, adrin: yes, i too vote for and savez options, but may be we should give a second round of check to see pickling in py3 and reading/loading in py2 works/does not work.19:01
adrinbesser82: we believe they are not segfaults and are the exceptions of python which are being interpreted by cmake as segfault. Can this be the case?19:01
snikbesser82: i am looking into the segfaults of all those Python_modulars, i will get back if can find some thing on it19:01
besser82adrin:  nope, those segfaults are segfaults  :(19:02
besser82adrin:  but not in python directly19:02
besser82adrin:  the python_modular just interfaces to libshogun19:02
besser82adrin:  and the segfault come from there...19:02
adrinbesser82: couldn't it be because the data is no loaded properly?19:02
besser82adrin:  I suspect the incompat between py3-ustr and char*19:03
besser82adrin:  py2::str == char*19:03
besser82adrin:  py3::ustr == somewhat homebrewed by python-devs19:03
besser82adrin:  but py3::bytes == char*  ;)19:04
adrinbesser82: oh yeah you're right. It perfectly makes sense if we're passing some str data to the C code, but if the data is just a numpy array and a bunch of numbers, why would it give a segfault?19:04
besser82adrin:  nope that just isn't numpy-only19:05
besser82adrin:  there is some json-style data19:05
besser82adrin:  which is stored as py2::str and get converted to py3::ustr on import ---> crash19:05
besser82adrin:  we need to find a way to store that json-string somehow both py-ver will read correctly19:06
besser82adrin:  but howss19:06
besser82adrin:  but how? ;)19:06
adrinbesser82: yeah it makes sense, I hate to have to change the C code to support the unicode19:06
adrinbesser82: the quick solution would be to convert the encoding of all the strings into bytearray before giving it to the C code, which is straightforward19:07
adrinbesser82: but I'm not sure if that's a desireable solution19:07
besser82adrin: mhhh19:07
besser82adrin: that would need some custom unpickler, right?!?19:07
adrinbesser82: it might need to feed the encoding parameter while unpickling only in py3, and maybe also converging the ustr to bytearray right after loading the data.19:09
adrinbessesr82: about the np.savez, I just checked and it seems the data saved in py3 can be perfectly loaded in py219:11
besser82adrin: as I said  :D  kk19:20
snikbesser82, adrin:  leaving office now, see you guys tomorrow! Will sleep over the problem, hopefully we will shoot it down soon! :)19:25
besser82snik:  allrighty!  Gute Nacht, sch?nen Feierabend  :D  Many thx for your help!19:25
snikbesser82:   my pleasure, if i can be of any help to the community! Danke, gleichfalls :)19:27
-!- thoralf [] has joined #shogun19:28
thoralfHeyhey :)19:28
besser82thoralf:  Hey!19:29
thoralfS?ren told you're missing me. ;)19:29
-!- iglesiasg [] has joined #shogun19:31
-!- mode/#shogun [+o iglesiasg] by ChanServ19:31
@iglesiasgsup people19:32
besser82thoralf:  yes, the stammtisch feels so empty w/o you  ;)19:32
besser82iglesiasg:  nuthang new ;)  And yourself?19:32
@iglesiasgbesser82, in process of git bisect-ing :)19:33
besser82iglesiasg:  Nice!  Sound promising to get that testsuite running  ;)19:33
-!- pickle27 [d8bfe7da@gateway/web/freenode/ip.] has joined #shogun19:45
@sonney2kiglesiasg, how is it going?19:56
@sonney2kAnd heya good evening gents :)19:57
@sonney2kpickle27, any news from across the world?19:57
pickle27not really!19:58
@iglesiasgsonney2k, compiling another revision :D19:58
pickle27had a question about gsoc - when is the app deadline?20:00
-!- votjakovr [~votjakovr@] has joined #shogun20:00
@iglesiasglet's check20:00
@iglesiasgthere you go20:01
@iglesiasgvalentine's day :DD20:01
besser82sonney2k:  heyya!20:01
besser82adrin:  ping?20:01
@sonney2kpickle27, verrry soon...20:01
pickle27cool! I was just curious - are we submitting all of those to the idea list or are we going to clean it up a bit20:01
adrinbesser82: ping-reply20:02
@sonney2khey votjakovr!20:02
besser82adrin:  wb  :D20:02
@sonney2kpickle27, we need to bring it into gsoc'13 shape20:02
adrinbesser82: had to rush for dinner, mensa-last-minute-food :D20:02
@iglesiasgI think there will be updates to the ideas list20:02
besser82lisitsyn, lisitsyn1:  AUSWEIS BITTE!  :-P20:02
votjakovrsonney2k: hi :)20:03
@sonney2kadrin, besser82 so regarding numpy and pickle - why not save things with numpy to a file - then read it in and pickle it as string / byte array?20:05
@sonney2kor snik ^^20:05
@iglesiasgpickle27, btw, I saw last week the video you posted on fb about shopify20:06
@iglesiasgpickle27, the company looks really great man!20:06
lisitsyn1besser82: mein ausweis ist weg20:06
pickle27yeah its a pretty good time here20:06
pickle27btw if anyone is looking for a job .. just let me know20:06
besser82lisitsyn1: :-P20:06
lisitsyn1pickle27: remote? ;)20:07
@iglesiasgpickle27, will ask in three years :D20:07
-!- parijat [671b082a@gateway/web/freenode/ip.] has joined #shogun20:07
pickle27we have some remote people, we mostly try and get people to relocate atm20:07
adrinsonny2k: I'm confused, if you save numpy data with, you can load it in another python version. About strings, we need to set the encoding while loading in py3, and maybe then change the encoding before passing to the C code20:07
besser82sonney2k:  as adrin said...20:08
pickle27lisitsyn1: I also joined the sublime text bandwagon20:08
besser82sonney2k:  numpy-arrays in pickles are not compat between py2 / py3 and never will20:08
lisitsyn1pickle27: cool, I think I am going to do that too20:08
lisitsyn1oh sh I just had 13 hrs workday after fulltime working weekend20:09
@sonney2kadrin, besser82 the point is we want a *single* .pickle file to write to20:09
lisitsyn1this makes no sense :D20:09
besser82sonney2k:  as it looks like atm: not a chance20:09
besser82sonney2k:  or we will definetively need an own impl. of Unpickler20:10
@sonney2kadrin, besser82 so we could do a trick to store numpy arrays - we save them to a file with save, read them in as ascii string - followed by some decoding20:10
besser82sonney2k:  that is not needed20:10
besser82sonney2k:  one can save that data with numpy's own methods and read it in as needed20:10
adrinsonny2k: I guess you're mixing two seperate problems, strings, and numpy arrays.20:10
besser82sonney2k:  the string-problem is mostly related to the segfaults, when restoring modshogun-data from pickles20:11
@sonney2kbesser82, but how do you then save shogun's objects and numpy arrays when they are in the same list?20:11
@sonney2kadrin, no no20:11
besser82sonney2k:  in different pickles  :-P20:12
besser82sonney2k:  one simply cannot have a pickle with that data to be compatible between py2 / py320:12
besser82sonney2k:  the other approach would be json or hdf520:13
besser82sonney2k:  hdf5 using pytables20:13
-!- thoralf [] has quit [Ping timeout: 265 seconds]20:14
besser82sonney2k:  the numpy-devs discussed that for ~3 monthes20:14
besser82sonney2k:  how to keep pickles compat20:14
besser82sonney2k:  but one cannot accomplish20:15
besser82sonney2k:  if there would have been a way they would have done it20:15
-!- thoralf [] has joined #shogun20:17
pickle27lisitsyn1: I thought you already did? we talked about it a while ago20:18
lisitsyn1pickle27: nah no20:18
lisitsyn1wanted to20:18
pickle27its pretty sweet, plus there is vintage mode if there is some vim command you want20:18
pickle27btw do we still use tabs in shogun?20:19
@sonney2kbesser82, ohh well - we could of course save numpy data as sth else but in an extra file - nah we have too many already and we could certainly save each element in the list we are returning into the same file with some minor logic20:24
@sonney2kbesser82, we could drop returning numpy objects altogether but instead work with scalars / the shogun objects directly20:25
@sonney2kpickle27, how is it going with the website?20:25
besser82sonney2k:  which still will not be pickable...20:25
pickle27I've been poking around20:25
pickle27its complicated because we want all that information just also to be simpler20:26
adrinbesser82: if the json dump for ndarray could work, why wouldn't numpy-devs use it?20:26
besser82adrin:  they do sth. similar in their load/save methods  ;)20:26
adrinbesser82: but anyway I'm not much in favor of dumping floating points as serialized-text data20:27
besser82adrin:  yes, that might be problematic20:27
@sonney2kbesser82, sure it would - look we don't need the all the magic20:33
@sonney2kbesser82, just in the examples we return instead of the numpy array20:33
@sonney2ksome string20:33
@sonney2kand that /can/ be pickled and compared20:34
-!- parijat [671b082a@gateway/web/freenode/ip.] has quit [Ping timeout: 272 seconds]20:34
@sonney2kor even we return the numpy array but iterate over objects if they are a list/tuple and convert them to some string representation - aka binary npy format20:35
-!- lambday [67157e37@gateway/web/freenode/ip.] has quit [Ping timeout: 272 seconds]20:35
@sonney2khey thoralf - I am still confused what to do with the bmrm fix /patch of yours20:36
thoralfsonney2k: Me too. ;)20:36
adrinsonney2k: so you're suggesting to make everything numpy-dump-able?20:37
thoralfsonney2k: I'm was pretty sure that I did the right thing, but Michal did the whole thing, so I need to check even more careful. ;)20:37
@sonney2kadrin, well we have the following setup: we return some list / tuple of results in our examples20:38
@sonney2kadrin, some of them are scalars, some SGObjects and some is numpy stuff20:39
@sonney2kadrin, in our regression tests we simply check if the objects are the same / similar20:39
@sonney2kso why not for numpy stuff add some extra treatment?20:39
@sonney2kthoralf, yeah to bad he didn't respond to your last Q20:40
adrinsonney2k: yeah sure, if it solves the problem it makes sense.20:42
thoralfsonney2k: Any suggestion how to proceed?20:42
thoralfI could reason (in more detail) about how I think it is, but maybe someone else could double-check my arguments...20:43
@sonney2kthoralf, well it is michael who knows - the only idea I have is take some big example train it with bmrm and then apply the patch and see if results are *exactly* the same20:44
@sonney2kiglesiasg, done with bisect?20:45
@iglesiasgsonney2k, Bisecting: 3 revisions left to test after this (roughly 2 steps)20:45
@iglesiasgI am going to see those revisions as well...20:45
@iglesiasgbut I am suspicious20:45
@iglesiasgall the revisions I have found so far during the process were good...20:46
adrinsonney2k, besser82: I need to catch the bus, see you tomorrow.20:47
@sonney2kadrin, cu!20:47
besser82adrin:  kk,  sch?nen Feierabend && Gute Nacht!20:47
thoralfsonney2k: But since his implementation is definitely wrong (producing memory errors when setting beta), I don't expect the same results.20:48
@sonney2kiglesiasg, if you figure that out the only remaining test is evaluation_clustering_simple20:48
@sonney2ktoday was a tough day in berlin - walking on ice20:49
@iglesiasgI am starting to get concerned that there is something wrong going on here20:49
thoralfsonney2k: I think the only question is if my patch is correct (beside the memory integrity).20:49
@sonney2kI've never seen more people fall...20:49
thoralfsonney2k: But you're idea is good: We could try to solve the same problem and then compare duality gap or something like that.20:52
@sonney2kthoralf, actually we have some test even20:53
@sonney2kjust since yesterday :)20:53
thoralfOr use the weight vector and use the (training) data to compute sth. like average square mean between predicted and trained value20:54
@sonney2kthoralf, no no - it should be 100% identical20:54
thoralfsonney2k: I think the implementation is wrong!20:55
thoralfThe beta is initialized wrong!20:55
thoralfSo why should it return the *same* results?20:55
@sonney2kthoralf, muahh then you should email vojtech / CC michael with your explanation. they are $DEITY20:56
lisitsyn1don't fix it20:56
lisitsyn1it is going to be a hydra20:57
-!- charry [~charry@] has joined #shogun20:57
@sonney2klisitsyn, yes marvin :)20:58
thoralflisitsyn1: I already fixed it. ;)20:58
thoralfsonney2k: vojtech-who?20:58
@sonney2kthoralf, vojtech franc20:59
thoralfHmm, okay.21:07
@iglesiasgsonney2k, I screwed up21:10
@iglesiasga revision that yesterday failed, now it doesn't fail21:11
@iglesiasgonly reason I found, because I am compiling using ccache and something is messing around with this21:11
@sonney2kiglesiasg, :`-(21:12
@iglesiasgyep, exactly21:12
@sonney2kiglesiasg, did you uninstall previous releases?21:12
@iglesiasgsonney2k, no21:12
@sonney2kperhaps you have a include conflict?21:13
@sonney2kiglesiasg, ohh you have to!21:13
@iglesiasghow do I do that?21:13
@iglesiasgI was basically21:13
@iglesiasggit co some_revision21:13
@iglesiasgccache make -j221:13
@iglesiasgsudo make install21:13
@iglesiasgin tests/integration/python_modular $ python linear_time_mmd.py21:13
-!- negativ [~negativ@] has joined #shogun21:14
@iglesiasgsonney2k, what should I have done?21:14
@iglesiasgsonney2k, rm -rf build each time? uninstall how?21:14
@sonney2kiglesiasg, always remove the installed shogun dir incl headers/libshogun/python stuff21:14
@sonney2kand rm-rf build too all the time21:14
@wikinginclude conflict should have been fixed already by one of besser82's commit21:15
-!- charry [~charry@] has quit [Ping timeout: 252 seconds]21:15
@sonney2kwiking, doesn't help if we git bisect old releases21:15
@wikingadding make uninstall target to cmake is a fairly easy process, there are howtos for it so if somebody is up for that should do it21:15
@wikingsonney2k: indeed21:16
@iglesiasgaargh fuck me man, this is so annoying21:16
@sonney2kand that is what we are doing21:16
@wikingsonney2k: bisect within a docker image21:16
@sonney2kwiking, how does that help?21:16
@sonney2kI mean rm'ing is far easier21:16
@wikingwell you can just delete the container...21:16
@wikingand start from the next commit21:16
@iglesiasgsonney2k, btw I am not sure what dirs yo umean with "remove the installed shogun dir incl headers/libshogun/python stuff"21:17
@wikingiglesiasg: e.g. /usr/local/include/shogun /usr/local/lib/libshogun* /usr/local/lib/python2.7/sites-package/shogun21:18
@wikingsomething like this21:18
@wikingif u have installed it21:18
@wikingof cours21:18
@iglesiasgwiking, got it, thanks!21:18
@iglesiasgwiking, I was not sure if there were those ones or something else inside shogun tree dir21:18
@wikingiglesiasg: but you dont have to make install to be able to run ctests21:18
@iglesiasgwiking, the problem is that in this case I cannot try it with ctest21:19
@iglesiasgwiking, I have to run by hand the tester21:19
@iglesiasgwiking, let me show you why21:19
@wikingiglesiasg: then just set PYTHONPATH21:19
@wikingand LD_LIBRARY_PATH21:19
@iglesiasgwhy so?21:19
@iglesiasgto what?21:19
@wikingso that u dont need the make install21:19
@wikingwell why else would you kill yourself with make install? :)21:19
@sonney2ktrue - you don't need make install then21:19
@wikingand then do rm -rf all the time after a check21:20
@iglesiasgdon't get it21:20
@iglesiasgI thought swig interfaces required the sudo make install21:20
@wikingiglesiasg: well u dont need to do make install in order to be able to do a simple check21:20
@wikingiglesiasg: no they dont21:20
@iglesiasgI guess the pythonpath ld_library_path thing fix that21:20
@wikingiglesiasg: what's the script u r trying to run?21:20
@iglesiasgwiking, I am running the integration test on linear_time_mmd.py21:21
@iglesiasgso tests/integration/python_modular/ linear_time_mmd.py21:21
@wikingwell you might have a smal problem21:21
@wikingwith the version21:21
@wikingwhere imports are not from modshogun21:21
@wikingbut from shogun21:21
@iglesiasgI think I am cool with doing install and removing the directories actually21:22
@iglesiasgjust would like to avoid repeating the process all over again :)21:22
@iglesiasgsonney2k, but I think it is better if I try to do this connecting remotely to fatbot21:24
@iglesiasgeach installation takes far too long locally21:24
@iglesiasghmm but will I have problems there with the installation issue??21:25
@sonney2kiglesiasg, don't install21:25
@sonney2klike wiking said21:25
@sonney2kit really is better21:25
@iglesiasgsonney2k, sure ok21:25
@iglesiasgbut tell me what to do instead21:25
@iglesiasgset PYTHONPATH, LD_LIBRARY_PATH21:25
@iglesiasgI don't really follow why that fixes it21:26
@iglesiasgI would understand that setting them tells make where to install the stuff21:26
@sonney2kiglesiasg, PYTHONPATH=path/to/modshogun ( the directory)21:26
@sonney2kiglesiasg, LD_LIBRARY_PATH=/path/to/libshogun (the dir again)21:26
@sonney2kand then python example.py21:26
@sonney2kiglesiasg, well if there is no default install of shogun -> no clash21:27
@iglesiasgsonney2k, the directory means from where I want to execute the python
-!- gsomix [~gsomix@] has joined #shogun21:28
@sonney2kiglesiasg, no the dir where the / libshogun*.so* are in21:28
@iglesiasgsonney2k, i am not sure what to do really21:30
@sonney2kiglesiasg, 1) remove shogun install'd files21:32
@sonney2k2) build again21:33
@sonney2k3) set python/ld_lib path21:33
@sonney2k4) run the example21:33
@sonney2kif any of this fails ask here21:33
gsomixsonney2k, hey. sry, I cannot to participate in today's hacking night. :( my laptop got wet by milk, need to dry.21:33
@iglesiasgaaaah I think I got it now21:33
@iglesiasgsonney2k, the key was telling me that 3) goes after 2) :)21:34
gsomixnothing serious, but keyboard does not work properly21:34
@sonney2kgsomix, you can type pretty well still ;)21:35
gsomixsonney2k, nope21:38
gsomixshift, ctrl, alt keys are crazy21:38
@iglesiasgsonney2k, was it ssh to for fatbot?21:41
@sonney2ksonney2k, fatbot.shogun-*21:42
@iglesiasgssh to is not working at all21:42
-!- infomix [~infomix@] has joined #shogun21:42
@sonney2kiglesiasg, works totally fine here21:43
@iglesiasgit sort of hangs up here21:43
@sonney2kiglesiasg, well then do it locally...21:43
@iglesiasgsonney2k, a little bit weird that I ping without a trouble though21:44
@sonney2klisitsyn, wiking can you still ssh fatbot?21:47
-!- gsomix [~gsomix@] has quit [Ping timeout: 272 seconds]21:47
@wikingsonney2k: i could about 4 hours ago21:48
@wikingand still i can21:49
@wiking15388 sonne     20   0 8135m  36m  392 R  99.6  0.1   7685:48 python21:49
@wiking15391 sonne     20   0 12.9g  36m  392 R  99.6  0.1   7726:18 python21:49
@wikingthey are running now for a while21:49
@iglesiasgwiking, would you mind adding a moment my public key?21:51
@iglesiasgwiking, the one that is there is an old one21:51
@wikingiglesiasg: in your home?21:51
@wikingssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDq1H9qVAPm7JZkhy+Mk2cwwExAJJgbnI0S9bT4ZO03caNWuBaP92c/mHNQi7x1+IyCCwd4jqInMrQ6MDTqz1YGZqsWknRzYxT0hs8Z1SSD06gs5APBfkaVpMgyuwLFyKTjqmib9H9vyZnTMj/d+ZYXxlb8DNT78aJqEnDRe8rNps4gCGOq5bJNkcaIV/aL5IJmwLmofjvhtD2O4I6U8pUgat6ulsJFlDTDCha3lEfMcP4KeqH7uSK/pCuT8Z1Xp0LaGgQjOlrHtTRJ73b3U5bXRKPMn+UmifdqXjIB5Vhbj0KDxnqcP3k8cJK27gUDXa8ApTj9oWOMFK6/qQxE/qLB fernando.iglesiasg@gmail.com21:52
@wikingthis is what you have in authorized_keys atm21:52
@wikingif u want something else21:52
@wikingand tell me if i should append or overwrite21:52
@iglesiasgwiking, let me see one second21:53
@wikingiglesiasg: u can actually give out public key in public.... that's the idea :P21:57
@wikingbut ok adding21:57
@wikingiglesiasg: tyr now21:57
@wikingshould be working21:57
@iglesiasgwiking, nope actually :S22:00
@wikingmmm interesting22:01
@wikingoh maaan22:02
@wikingyour problem is the username. u r trying with iglesiasg but your username on fatbot is iglesias22:02
@wikingwithout the last g on the end22:02
@iglesiasgwiking,  I tried iglesias too ;)22:03
@iglesiasgssh iglesias@fatbot.shogun-toolbox.7nn.de22:03
@iglesiasgPermission denied (publickey).22:03
@wikingplz try now again22:03
@wikingand and u r trying the wrong host as well22:03
@wikingssh iglesias@fatbot.shogun-toolbox.org22:04
@wikingu have 2 more seconds to try as after that i'm gone22:06
@wikingiglesiasg: if u want access to 7nn.de22:06
@wikingi can add there your public key as well22:06
@wikingno actually there i cannot22:06
@wikingas there i dont have sudo22:06
-!- thoralf [] has quit [Quit: Konversation terminated!]22:07
@iglesiasgwiking, .org worked, thanks!22:07
-!- votjakovr [~votjakovr@] has quit [Quit: WeeChat 0.4.0]22:07
@sonney2kohh man22:07
@sonney2kiglesiasg, now finally figure it out :)22:08
-!- infomix [~infomix@] has quit [K-Lined]22:18
-!- negativ [~negativ@] has quit [K-Lined]22:18
-!- rymrym [~rymrym@] has joined #shogun22:21
-!- pickle27 [d8bfe7da@gateway/web/freenode/ip.] has quit [Ping timeout: 272 seconds]22:35
-!- shogun-notifier- [] has joined #shogun22:37
shogun-notifier-shogun-data: Soeren Sonnenburg :master * 6b289bd / testsuite/tests/ (2 files):
shogun-notifier-shogun-data: update data for fixed clustering evalution22:37
shogun-notifier-shogun: Soeren Sonnenburg :develop * 2483b5a / / (3 files):
shogun-notifier-shogun: fix ClusteringMutualInformation - init matrix as zero fixing valgrind errors and tests22:37
@sonney2kfinally - iglesiasg so now only the mmd stuff remains!22:38
-!- pickle27 [d8bfe7da@gateway/web/freenode/ip.] has joined #shogun22:38
@iglesiasgsonney2k, I am doing my best!22:40
@iglesiasgsonney2k, the process is much faster now compiling in fatbot22:42
@iglesiasgsonney2k, I hope it will work out this time to detect the revision that broke the test22:42
besser82sonney2k:  How do we want to get that py_mod SGObj thing started?22:43
@sonney2kbesser82, no idea22:43
@sonney2kbesser82, maybe gsoc task22:44
besser82sonney2k:  and Who to mentor?22:44
@sonney2ktoo big I am afraid22:44
besser82sonney2k:  what about that cmake-stuff && gsoc?22:44
besser82sonney2k:  shall I put that to iglesia's gdrive?22:44
@sonney2kbesser82, sure!22:48
besser82sonney2k:  kk22:48
@sonney2kbesser82, I shouldn't be mentoring more than 2 things so not sure if examples is a good thing for me - though I have a plan... the only problem to decide is whether we want some intermediate language to write the examples in or not22:49
besser82sonney2k:  not really,  keeping the examples lang by lang would be good22:53
besser82sonney2k:  just sharing the reference data22:53
@sonney2kbesser82, no no22:56
@sonney2kit is too much effort to write the examples lang by lang22:56
@sonney2kand they should be very simple just for reference22:56
@sonney2kmore complex stuff should be a notebook22:56
-!- rymrym [~rymrym@] has quit [K-Lined]22:57
besser82sonney2k:  ok, ok23:00
besser82sonney2k:  but what "intermediate" language to chose?23:01
@sonney2kno idea23:01
@sonney2kmaybe wiking has an idea?23:02
lisitsyn1w00t you are toalking abouuut?23:03
@sonney2klisitsyn, writing shogun examples in some intermediate language / or jinja templates so that it can easily be converted to all $LANG we support23:04
lisitsyn1sonney2k: think that's possible?23:05
lisitsyn1sounds ok but I have no idea23:05
lisitsyn1sonney2k: then we can follow common naming schemes!23:05
lisitsyn1e.g. it is just wrong to have get_stuff in java23:06
lisitsyn1nobody uses that convention in java23:06
lisitsyn1code looks un-idiomatic23:06
@sonney2klisitsyn, sure - if you look at the examples they all look the same except for some very minor deviations23:06
@sonney2klisitsyn, yeah swig supports that even...23:06
lisitsyn1hmm then we should have it23:07
lisitsyn1it must be really confusing to use _ in java23:07
@sonney2klisitsyn, yeah ... doing examples just once would have some huge advantage too23:09
pickle27I could be okay to mentor that project if besser82 has enough on his plate23:09
@sonney2klisitsyn, we could return only shogun objects23:09
@sonney2klisitsyn, and then these can be serialized with SerializableAsciiFile23:10
@sonney2kand read back in and compared with equals23:10
@sonney2kso we have cross language tests23:10
@sonney2kand all examples do the same  / it is ensured that they all give the same correct output23:10
@sonney2kpickle27, didn't you have sth else already?23:11
shogun-buildbotbuild #37 of deb4 - python3 is complete: Failure [failed test python modular]  Build details are at  blamelist: Soeren Sonnenburg <>23:11
pickle27I have the opencv integration23:11
pickle27I only want to mentor one project but if no one picks that one then i'd be happy to help on examples23:11
@sonney2kpickle27, OK23:13
@sonney2kpickle27, any idea what intermediate language one could use?23:13
@sonney2kI would want to keep it as simple as possible23:13
@sonney2kno idea if jinja2 templates would do...23:14
pickle27honestly most of the examples are already super basic because of the load helper methods23:15
@sonney2kpickle27, yes they are intended to be only reference examples23:15
pickle27I would probably do a second take and move all of the data generation into a contained class23:15
@sonney2kpickle27, we would want to make things even more simple / reference like23:15
@sonney2kpickle27, and put all the higher level stuff into notebooks etc23:15
@sonney2kpickle27, I think jinja2 is sufficient - I mean the shogun stuff has the very same syntax across languages23:17
@sonney2kand data is supposed to be loaded from disk likely even with shogun.23:17
pickle27yeah I think that would work23:17
@sonney2kpickle27, me might want to have a few examples still using this loader crap23:17
@sonney2kbut that again has the same syntax23:17
@sonney2kpickle27, problem is if we pose this for a student we should try *one* minimal example - like e.g. compute the Gaussian Kernel or so23:18
pickle27I would make all the examples use the loader23:18
pickle27then it would be super simple to generate them23:19
@sonney2kpickle27, why not - I think we could have some cmake rule that converts *.py.jinja2 into *.py23:20
@sonney2kand if other *.py's exist it would just not touch things23:20
@sonney2kso we *could* have additional examples if stuff doesn't work out23:20
pickle27yeah totally23:22
pickle27btw just to be clear I'm talking about making sure all these are consistent and then use them to gen examples:
@sonney2kpickle27, yes23:23
@sonney2kpickle27, we are still talking about hundreds of examples23:23
pickle27okay just checking we're on the same page23:23
@sonney2kso it will be quite some pile of work23:24
pickle27yeah i know23:24
pickle27there is probably lots that we don't really need slash are basically repeated from unit tests23:24
@sonney2kalso they need to be changed to just return shogun objects - e.g. the kernel's can only be reasonably tested when they return the kernel matrix23:25
@sonney2kbut that would be an SGMatrix - so we need to put them in some CustomKernel or have some other datatype for that that has a .equals23:26
pickle27actually just thinking now, it might be cool to try and move a lot of those really simple tests to be automatically copied from unit tests23:27
pickle27cause we're packing a lot of duplication right now aren't we?23:27
pickle27I know that all of the examples I wrote for gsoc were basically the same as my unit tests23:28
pickle27we could have a flag or something that says this unit test is also a basic example and then it gets copied in23:28
pickle27anyways I got to run but I might look into that after Im done on the website see if I can put something together23:29
-!- pickle27 [d8bfe7da@gateway/web/freenode/ip.] has quit [Quit: Page closed]23:30
@sonney2kpickle27 it is actually even more simple - a unit test would compute e.g. the gaussian kernel and check if values are as expected - in the example you just have the GaussianKernel and done23:31
shogun-buildbotbuild #2142 of deb3 - modular_interfaces is complete: Failure [failed test python modular]  Build details are at  blamelist: Soeren Sonnenburg <>23:47
--- Log closed Tue Jan 21 00:00:01 2014