Open in new window / Try shogun cloud
--- Log opened Mon Jan 20 00:00:00 2014
-!- shogun-notifier- [~irker@7nn.de] has quit [Quit: transmission timeout]00:59
-!- sonne|osx_ [~sonne@f052020014.adsl.alicedsl.de] has joined #shogun03:29
-!- sonne|osx [~sonne@f052018132.adsl.alicedsl.de] has quit [Ping timeout: 272 seconds]03:31
-!- sonne|osx_ is now known as sonne|osx03:31
shogun-buildbotbuild #686 of nightly_default is complete: Failure [failed test]  Build details are at http://buildbot.shogun-toolbox.org/builders/nightly_default/builds/68604:12
-!- shallhome [~shallhome@193.28.144.96] has joined #shogun04:40
-!- shallhome [~shallhome@193.28.144.96] has quit [Remote host closed the connection]05:14
-!- sonne|osx [~sonne@f052020014.adsl.alicedsl.de] has quit [Quit: sonne|osx]07:21
-!- sonne|osx [~sonne@89.204.130.57] has joined #shogun07:49
sonne|osxbesser82: we have the same pickle problem again07:54
sonne|osxbesser82: please enlighten me how you fixed it!07:54
-!- lisitsyn [~lisitsin@mxs.kg.ru] has joined #shogun07:56
-!- sonne|osx [~sonne@89.204.130.57] has quit [Ping timeout: 252 seconds]08:00
-!- sonne|osx [~sonne@89.204.137.33] has joined #shogun08:04
-!- sonne|osx [~sonne@89.204.137.33] has quit [Client Quit]08:05
-!- sonne|osx [~sonne@24-134-74-216-dynip.superkabel.de] has joined #shogun08:19
-!- sonne|osx [~sonne@24-134-74-216-dynip.superkabel.de] has quit [Client Quit]08:21
besser82sonne|work:  which pickle-problem @ which test / build?  I can haz linxx, plz?09:43
sonne|workbesser82: well all python3 tests fail due to me adding  the pickle string based comparison09:46
sonne|workbesser82: so IIRC you figured there is some encoding problem....09:47
besser82sonne|work:  encoding ain't the only one  ;)09:47
besser82sonne|work:  but in fact the major one09:47
besser82sonne|work:  lemme cycle through all recent commits to see what changed and I'll give you some advice  ;)09:48
sonne|workbesser82: well it is just this pickle.dumps(objs) == comparison in tester.py09:50
besser82sonne|work:  then there is no general solution (tm);  the most problematic thing is the incompat between numpy and py3-numpy, which can't be solved without external workaround like storing the numpy-datatypes with numpy's internal serialization methods, unfortunately.  :(10:41
sonne|workbesser82: well all I want is that this doesn't crash!10:41
besser82sonne|work:  you mean the segfaults, don't you?10:42
sonne|workyes10:42
besser82sonne|work:  those were never fixed IIRC10:42
-!- lambday [67157f36@gateway/web/freenode/ip.103.21.127.54] has joined #shogun10:43
sonne|workbesser82: I thought you did sth with the encoding?10:43
besser82sonne|work:  because of we cannot restore py2_mod stuff with py3_mod...10:43
besser82sonne|work:  I did some experiments with that, but that actually wasn't successful  :(10:44
besser82sonne|work:  It's not only the encoding / datatypes used in pickle; it's basically more  :(10:44
besser82sonne|work:  I tried sth. with manually hex-editing the pickles to have str-data stored as py3-bytes, but then there is some other serious trouble, when opening them with py3_modular  :(10:46
besser82sonne|work: the only way I see to get that fixed is to have different pickles for py2 and py310:48
besser82sonne|work: but hoding the same data...10:48
besser82sonne|work:  which brings us to the next question:  How to get that done?10:49
besser82sonne|work:  I mean, those pickled must have been created somewhen; but how?10:49
sonne|workbesser82: well it is just some ascii file that we create11:07
sonne|workbesser82: how would embed some ascii string python2/3 compatible11:07
adrinsonne|work, besser82: hey, what's up? Good morning.11:24
sonne|workadrin: good morning11:24
besser82sonne|work:  basically I would try to use some py3::bytes-type which is perfectly binary-compat with char*11:24
sonne|workadrin: we are trying to get python3 to work ...11:24
besser82sonne|work:  but the problem here is py2 doesn't know "bytes" and has str only, which is the same as char*, too11:25
adrinif you put a "b" before the string, it will be interpreted as a byte-char in py3, and will be ignored in py2.7, why doesn't that work here?11:25
sonne|workadrin: who says it wouldn't :)11:26
besser82adrin:  Hey! G'Mornin!  Yes, as sonne|work says all about Python3 && shogun11:26
sonne|workadrin: we are just not python3 experts11:26
sonne|workthat is something we can fix at tonights stammtisch :)11:26
adrinbesser82, sonne|work: there are two prefixes for strings in python, "u" and "b"11:26
sonne|workadrin: in python 2.6 etc too?11:26
adrin"u" is ignored by py3, and "b" is ignored by 2.711:27
adrinno py2.6 doesn't know about them11:27
adrinsonne|work: these two were introduced in py2.7, to make people able to have strings compatible in both 2 and 3 pythons, but you will need to have at least py2.711:27
besser82sonne|work, adrin:  for Py2.6 one needs to `import __future__` within some exception11:28
besser82sonne|work, adrin:  like try: import __future__ except: #noop11:28
sonne|workadrin: how do we get these if we are reading a file?11:28
besser82sonne|work, adrin:  but that won't fix the problem with pickles  :-P11:29
adrinsonne|work: so the problem is that you want to read a file and you need to make sure you're interpreting it as ASCII bytes?11:29
besser82adrin: no, we are importing a pickle made with proto 211:30
besser82adrin: and for some reason py3_modular segfaults when restoring the py2_mod data from that pickle  ;)11:30
sonne|worksry no time for that jsut now11:30
sonne|workadrin: will you be around in the evening?11:30
adrinsonne|work: yeah sure. I'll try to see what I can do until then.11:31
adrinbesser82: In the mean time, it would be nice if you could send me a py2_mod data so that I can try to load it.11:38
besser82adrin:  you can fetch them from here: https://github.com/shogun-toolbox/shogun-data/tree/master/testsuite/tests11:39
besser82adrin:  here is a list of those failing / segfaulting tests:  http://buildbot.shogun-toolbox.org/builders/deb4%20-%20python3/builds/3611:39
adrinbesser82: thanks11:40
besser82adrin:  the BIG problems is the binary-incompat of numpy-arrays && those segfaults, when restoring data from python_modular of shogun  ;)  np.  You're welcome  :D11:40
-!- lambday [67157f36@gateway/web/freenode/ip.103.21.127.54] has quit [Ping timeout: 272 seconds]11:41
adrinbesser82: I see11:41
besser82adrin: btw. manually converting str to bytes in pickle didn't work.  :(11:42
besser82adrin: like sed -e 's!^S\'!B\'!g' pickle.file11:43
-!- bb_ [458fc8de@gateway/web/freenode/ip.69.143.200.222] has joined #shogun15:19
-!- bb_ [458fc8de@gateway/web/freenode/ip.69.143.200.222] has quit [Ping timeout: 272 seconds]15:39
-!- sonne|osx [~sonne@89.204.138.241] has joined #shogun15:40
sonne|osxadrin, besser82 btw we can of course generate the data again with python2.7 or python3 if that helps15:46
besser82sonne|osx:  that would be the easiest solution  ;)15:46
besser82sonne|osx:  we need separate pickles for py2 && py3 and all would be fine15:47
sonne|osxbesser82: would this ensure that it can be read with python2.7 then?15:47
sonne|osxbesser82: nah15:47
sonne|osxone for all15:47
besser82sonne|osx:  one for all isn't possible ---> numpy15:47
sonne|osxbesser82: so there is no way to load numpy data from python3?15:47
besser82sonne|osx:  it is when pickle is from py315:48
besser82sonne|osx:  they are not interchangeable15:48
besser82sonne|osx:  that's why numpy has own im/export methods for data  ;)15:48
sonne|osxbesser82: and stuff like pickle.load(file_obj, encoding='latin1') ?15:48
besser82sonne|osx:  will be readble, but numpy data is a mess then15:49
besser82sonne|osx:  as to be seen in tesstsuite on py3, which fail but don't segfault  ;)15:49
sonne|osxbesser82: hmmhh btw will you be around for the stammtisch tonight?15:49
besser82sonne|osx:  sure!15:49
besser82sonne|osx:  not till the end, but at least ~2 h15:50
sonne|osxyeah sure15:51
sonne|osxbesser82: no idea why the numpy people did it this way15:52
sonne|osxsucks badly that you cannot load your old data15:52
besser82sonne|osx:  you can but only with some extra steps15:52
sonne|osxbesser82: which are?15:52
besser82sonne|osx:  you need to load from the py-ver the pickle was made with and then export the numpy-data with the export-method of numpy15:53
besser82sonne|osx:  that export is gurateed to be readable from every recent python15:53
sonne|osxbesser82: which would be savenpy? or what15:53
sonne|osxc'mon15:54
sonne|osx:(15:54
besser82sonne|osx:  load pickle15:55
besser82sonne|osx:  $my_numpy_array.save($file_handle)15:55
besser82sonne|osx:  open other python-ver15:56
besser82sonne|osx:  create empty numpy-array15:56
besser82sonne|osx:  $new_array.load($my_export_from_old_pyver)15:56
sonne|osxbesser82: not good - maybe we can work around this with python's stringIO15:57
sonne|osxadrin: what do you think? Maybe we store numpy stuff with stringIO - then we could keep one single format?15:58
besser82sonne|osx: why using stringIO?15:58
sonne|osxbesser82: string that behaves like a file handle15:58
besser82sonne|osx: i mean numpy has own methods for that... ???15:59
sonne|osxbesser82: so we could write that string out instead of pickle15:59
sonne|osxand it would be compatible15:59
besser82sonne|osx: that can be done with those numpy.methods, too15:59
besser82sonne|osx: those are gurateed to be compat within different python-interp16:00
sonne|osxbesser82: lets talk about that in the evening16:00
besser82sonne|osx:  kk16:00
* sonne|osx off train16:00
-!- sonne|osx [~sonne@89.204.138.241] has quit [Quit: sonne|osx]16:00
-!- lambday [67157e37@gateway/web/freenode/ip.103.21.126.55] has joined #shogun16:04
-!- parijat [671b082a@gateway/web/freenode/ip.103.27.8.42] has joined #shogun16:10
adrinbesser82: me and my office mate were looking at the errors/data16:11
besser82adrin:  Many thx!  and what did you find out?16:12
adrinbesser82: he's never had such a problem with pickle, but he never used different python versions16:12
besser82adrin:  same for us, too  ;)16:12
adrinbesser82: nothing yet, but we're trying to see how to tell pickle how to interpret strings16:13
besser82adrin:  thanks for your work  ;)16:13
adrinbesser82: I personally prefer to handle it that way, or generating the data using utf strings, and tell pickle in py2 to interpret them as utf, I guess then there is no need for multiple data files, right?16:14
besser82adrin:  would that work with numpy-arrays?16:14
adrinbesser82: don't know yet, but we have py2.6, 2.7, and 3.3 available. Personally I don't recommend to worry/support 2.6. 2.7 and 3.3 should be enough16:16
besser82adrin:  I would apprechiate support for py2.6 because of RHEL6  ;)16:16
besser82adrin:  But those details i might add somewhen later then  ;)16:16
adrinbesser82: I see. Then we'll test 2.6 too. Shall I ask for a peice of code which generates the pickle data + a sample saved data + the code which is supposed to read it? We found many files which can't read a data, but it would be nice if we had just an example, if available. If not, we'll try with a simple string/np.ndarray/set/list file if even with such a simple example the problem perists.16:19
besser82adrin:  I think sonney2k has that code or some examples about generation....16:21
besser82adrin:  but trying with some toy data would at least point into the right direction, I think.16:21
adrinbesser82: ok, we'll do it. BTW, is the stammtisch a real physical one or you guys were talking about a virtual stammtisch? :D16:23
-!- gsomix [~gsomix@46.20.65.64] has quit [Ping timeout: 264 seconds]16:28
besser82adrin:  virtual one on irc  ;)16:33
besser82adrin:  distances are a bit problematic  ;)16:33
besser82adrin:  you, toralf and sonney2k possibly could meet physically Berlin  ;)16:34
besser82adrin:  me is from Hildesheim (30 km south of HAnnover)16:34
-!- snik [~snik@lap-13-03.ag3.mpi-sb.mpg.de] has joined #shogun16:37
besser82snik:  hi!  You must be one of adrin's office mates, right?!?16:37
snikHi! besser82, yes, i am!16:38
besser82snik:  nice to meet you!  :D16:39
snikNice to meet you too! :)16:39
besser82snik:  and thank for helping with that py3-thing!  :D16:39
adrinbesser82: haha, we're based in Saarbruecken, Berlin is a bit far :D16:39
snikoh, we have just started..16:39
besser82adrin:  kk, that's a bit far to Berlin, too  ;)16:40
adrinbesser82: why the 'latin_1' decoding option for numpy arrays is not an option here? I didn't understand why it's not desireable.17:53
besser82adrin:  the problem is: numpy-array-pickles are not binary compatible between py2 and py3  :(17:54
besser82adrin:  one can encode / decode the pickled-data using 'latin-1', but the outcome is just garbage17:55
adrinbesser82: ok, let me see, it didn't seem garbage, let me check17:56
adrinbesser82: my mistake, it IS garbage. will be back soon :D17:57
snikok, right, it really is!17:57
besser82adrin, snik:  numpy.save/load() works between pyvers, but pickle seems not.  :(17:58
-!- genrut [~genrut@193.28.144.131] has joined #shogun17:59
snikbesser82: pickle is really an issue, visibly even for the larger community, especially with the protocol thingy18:00
besser82snik:  unfortunately, yes  :(18:00
-!- genrut [~genrut@193.28.144.131] has quit [Ping timeout: 246 seconds]18:09
-!- parijat [671b082a@gateway/web/freenode/ip.103.27.8.42] has quit [Ping timeout: 272 seconds]18:40
adrinbesser82: I guess we failed :D, the problem is the ndarray representation is in unicode format, and can not be read in python 218:56
adrinbesser82: we tried to save using py3, and load in py2, but for this reason it doesn't work18:57
besser82adrin:  save like pickle?18:57
besser82adrin:  or numpy.save?18:57
adrinbesser82: yeah, in theory it should work if the protocol is 2, but it doesn't18:57
adrinbesser82: did you guys try the numpy.save/savez? I would vote for them if they work between python versions18:58
adrinbesser82: we thought if we could save the data in py3 in a way that py2 could read it, then somehow the problem is solved, but even this solution didn't work18:59
besser82adrin:  let me discuss that with sonney2k on tonight's stammtisch...18:59
besser82adrin: the other ugly thing is those segfaults of python3_modular, when trying to recreate data from py2 pickle...  :(19:00
besser82adrin:  any thoughts on that?19:00
snikbesser82, adrin: yes, i too vote for numpy.save and savez options, but may be we should give a second round of check to see pickling in py3 and reading/loading in py2 works/does not work.19:01
adrinbesser82: we believe they are not segfaults and are the exceptions of python which are being interpreted by cmake as segfault. Can this be the case?19:01
snikbesser82: i am looking into the segfaults of all those Python_modulars, i will get back if can find some thing on it19:01
besser82adrin:  nope, those segfaults are segfaults  :(19:02
besser82adrin:  but not in python directly19:02
besser82adrin:  the python_modular just interfaces to libshogun19:02
besser82adrin:  and the segfault come from there...19:02
adrinbesser82: couldn't it be because the data is no loaded properly?19:02
besser82adrin:  I suspect the incompat between py3-ustr and char*19:03
besser82adrin:  py2::str == char*19:03
besser82adrin:  py3::ustr == somewhat homebrewed by python-devs19:03
besser82adrin:  but py3::bytes == char*  ;)19:04
adrinbesser82: oh yeah you're right. It perfectly makes sense if we're passing some str data to the C code, but if the data is just a numpy array and a bunch of numbers, why would it give a segfault?19:04
besser82adrin:  nope that just isn't numpy-only19:05
besser82adrin:  there is some json-style data19:05
besser82adrin:  which is stored as py2::str and get converted to py3::ustr on import ---> crash19:05
besser82adrin:  we need to find a way to store that json-string somehow both py-ver will read correctly19:06
besser82adrin:  but howss19:06
besser82adrin:  but how? ;)19:06
adrinbesser82: yeah it makes sense, I hate to have to change the C code to support the unicode19:06
adrinbesser82: the quick solution would be to convert the encoding of all the strings into bytearray before giving it to the C code, which is straightforward19:07
adrinbesser82: but I'm not sure if that's a desireable solution19:07
besser82adrin: mhhh19:07
besser82adrin: that would need some custom unpickler, right?!?19:07
adrinbesser82: it might need to feed the encoding parameter while unpickling only in py3, and maybe also converging the ustr to bytearray right after loading the data.19:09
adrinbessesr82: about the np.savez, I just checked and it seems the data saved in py3 can be perfectly loaded in py219:11
adrinbesser82:brb19:12
besser82adrin: as I said  :D  kk19:20
snikbesser82, adrin:  leaving office now, see you guys tomorrow! Will sleep over the problem, hopefully we will shoot it down soon! :)19:25
besser82snik:  allrighty!  Gute Nacht, sch?nen Feierabend  :D  Many thx for your help!19:25
snikbesser82:   my pleasure, if i can be of any help to the community! Danke, gleichfalls :)19:27
-!- thoralf [~thoralf@91-65-136-3-dynip.superkabel.de] has joined #shogun19:28
thoralfHeyhey :)19:28
besser82thoralf:  Hey!19:29
thoralfS?ren told you're missing me. ;)19:29
-!- iglesiasg [~iglesiasg@524AE0A7.cm-4-3d.dynamic.ziggo.nl] has joined #shogun19:31
-!- mode/#shogun [+o iglesiasg] by ChanServ19:31
@iglesiasgsup people19:32
besser82thoralf:  yes, the stammtisch feels so empty w/o you  ;)19:32
besser82iglesiasg:  nuthang new ;)  And yourself?19:32
@iglesiasgbesser82, in process of git bisect-ing :)19:33
besser82iglesiasg:  Nice!  Sound promising to get that testsuite running  ;)19:33
-!- pickle27 [d8bfe7da@gateway/web/freenode/ip.216.191.231.218] has joined #shogun19:45
@sonney2kiglesiasg, how is it going?19:56
@sonney2kAnd heya good evening gents :)19:57
@sonney2kpickle27, any news from across the world?19:57
pickle27not really!19:58
@iglesiasgsonney2k, compiling another revision :D19:58
pickle27had a question about gsoc - when is the app deadline?20:00
-!- votjakovr [~votjakovr@188.134.46.30] has joined #shogun20:00
@iglesiasglet's check20:00
@iglesiasgpickle27, http://www.google-melange.com/gsoc/events/google/gsoc201420:01
@iglesiasgthere you go20:01
@iglesiasgvalentine's day :DD20:01
besser82sonney2k:  heyya!20:01
besser82adrin:  ping?20:01
@sonney2kpickle27, verrry soon...20:01
pickle27cool! I was just curious - are we submitting all of those to the idea list or are we going to clean it up a bit20:01
adrinbesser82: ping-reply20:02
@sonney2khey votjakovr!20:02
besser82adrin:  wb  :D20:02
@sonney2kpickle27, we need to bring it into gsoc'13 shape20:02
adrinbesser82: had to rush for dinner, mensa-last-minute-food :D20:02
@iglesiasgI think there will be updates to the ideas list20:02
besser82lisitsyn, lisitsyn1:  AUSWEIS BITTE!  :-P20:02
votjakovrsonney2k: hi :)20:03
@sonney2kadrin, besser82 so regarding numpy and pickle - why not save things with numpy to a file - then read it in and pickle it as string / byte array?20:05
@sonney2kor snik ^^20:05
@iglesiasgpickle27, btw, I saw last week the video you posted on fb about shopify20:06
@iglesiasgpickle27, the company looks really great man!20:06
lisitsyn1besser82: mein ausweis ist weg20:06
pickle27yeah its a pretty good time here20:06
@iglesiasgawesome20:06
pickle27btw if anyone is looking for a job .. just let me know20:06
besser82lisitsyn1: :-P20:06
lisitsyn1pickle27: remote? ;)20:07
@iglesiasgpickle27, will ask in three years :D20:07
-!- parijat [671b082a@gateway/web/freenode/ip.103.27.8.42] has joined #shogun20:07
pickle27we have some remote people, we mostly try and get people to relocate atm20:07
adrinsonny2k: I'm confused, if you save numpy data with numpy.save/savez, you can load it in another python version. About strings, we need to set the encoding while loading in py3, and maybe then change the encoding before passing to the C code20:07
besser82sonney2k:  as adrin said...20:08
pickle27lisitsyn1: I also joined the sublime text bandwagon20:08
besser82sonney2k:  numpy-arrays in pickles are not compat between py2 / py3 and never will20:08
lisitsyn1pickle27: cool, I think I am going to do that too20:08
lisitsyn1oh sh I just had 13 hrs workday after fulltime working weekend20:09
@sonney2kadrin, besser82 the point is we want a *single* .pickle file to write to20:09
lisitsyn1this makes no sense :D20:09
besser82sonney2k:  as it looks like atm: not a chance20:09
besser82sonney2k:  or we will definetively need an own impl. of Unpickler20:10
@sonney2kadrin, besser82 so we could do a trick to store numpy arrays - we save them to a file with save, read them in as ascii string - followed by some decoding20:10
besser82sonney2k:  that is not needed20:10
besser82sonney2k:  one can save that data with numpy's own methods and read it in as needed20:10
adrinsonny2k: I guess you're mixing two seperate problems, strings, and numpy arrays.20:10
besser82sonney2k:  the string-problem is mostly related to the segfaults, when restoring modshogun-data from pickles20:11
@sonney2kbesser82, but how do you then save shogun's objects and numpy arrays when they are in the same list?20:11
@sonney2kadrin, no no20:11
besser82sonney2k:  in different pickles  :-P20:12
besser82sonney2k:  one simply cannot have a pickle with that data to be compatible between py2 / py320:12
besser82sonney2k:  the other approach would be json or hdf520:13
besser82sonney2k:  hdf5 using pytables20:13
-!- thoralf [~thoralf@91-65-136-3-dynip.superkabel.de] has quit [Ping timeout: 265 seconds]20:14
besser82sonney2k:  the numpy-devs discussed that for ~3 monthes20:14
besser82sonney2k:  how to keep pickles compat20:14
besser82sonney2k:  but one cannot accomplish20:15
besser82sonney2k:  if there would have been a way they would have done it20:15
-!- thoralf [~thoralf@91-65-136-3-dynip.superkabel.de] has joined #shogun20:17
pickle27lisitsyn1: I thought you already did? we talked about it a while ago20:18
lisitsyn1pickle27: nah no20:18
lisitsyn1wanted to20:18
pickle27its pretty sweet, plus there is vintage mode if there is some vim command you want20:18
pickle27btw do we still use tabs in shogun?20:19
@sonney2kbesser82, ohh well - we could of course save numpy data as sth else but in an extra file - nah we have too many already and we could certainly save each element in the list we are returning into the same file with some minor logic20:24
@sonney2kbesser82, we could drop returning numpy objects altogether but instead work with scalars / the shogun objects directly20:25
@sonney2kpickle27, how is it going with the website?20:25
besser82sonney2k:  which still will not be pickable...20:25
pickle27I've been poking around20:25
pickle27its complicated because we want all that information just also to be simpler20:26
adrinbesser82: if the json dump for ndarray could work, why wouldn't numpy-devs use it?20:26
besser82adrin:  they do sth. similar in their load/save methods  ;)20:26
adrinbesser82: but anyway I'm not much in favor of dumping floating points as serialized-text data20:27
besser82adrin:  yes, that might be problematic20:27
@sonney2kbesser82, sure it would - look we don't need the all the magic20:33
@sonney2kbesser82, just in the examples we return instead of the numpy array20:33
@sonney2ksome string20:33
@sonney2kand that /can/ be pickled and compared20:34
-!- parijat [671b082a@gateway/web/freenode/ip.103.27.8.42] has quit [Ping timeout: 272 seconds]20:34
@sonney2kor even we return the numpy array but iterate over objects if they are a list/tuple and convert them to some string representation - aka binary npy format20:35
-!- lambday [67157e37@gateway/web/freenode/ip.103.21.126.55] has quit [Ping timeout: 272 seconds]20:35
@sonney2khey thoralf - I am still confused what to do with the bmrm fix /patch of yours20:36
thoralfsonney2k: Me too. ;)20:36
adrinsonney2k: so you're suggesting to make everything numpy-dump-able?20:37
thoralfsonney2k: I'm was pretty sure that I did the right thing, but Michal did the whole thing, so I need to check even more careful. ;)20:37
@sonney2kadrin, well we have the following setup: we return some list / tuple of results in our examples20:38
@sonney2kadrin, some of them are scalars, some SGObjects and some is numpy stuff20:39
@sonney2kadrin, in our regression tests we simply check if the objects are the same / similar20:39
@sonney2kso why not for numpy stuff add some extra treatment?20:39
@sonney2kthoralf, yeah to bad he didn't respond to your last Q20:40
adrinsonney2k: yeah sure, if it solves the problem it makes sense.20:42
thoralfsonney2k: Any suggestion how to proceed?20:42
thoralfI could reason (in more detail) about how I think it is, but maybe someone else could double-check my arguments...20:43
@sonney2kthoralf, well it is michael who knows - the only idea I have is take some big example train it with bmrm and then apply the patch and see if results are *exactly* the same20:44
@sonney2kiglesiasg, done with bisect?20:45
@iglesiasgsonney2k, Bisecting: 3 revisions left to test after this (roughly 2 steps)20:45
@iglesiasgI am going to see those revisions as well...20:45
@iglesiasgbut I am suspicious20:45
@iglesiasgall the revisions I have found so far during the process were good...20:46
adrinsonney2k, besser82: I need to catch the bus, see you tomorrow.20:47
@sonney2kadrin, cu!20:47
besser82adrin:  kk,  sch?nen Feierabend && Gute Nacht!20:47
thoralfsonney2k: But since his implementation is definitely wrong (producing memory errors when setting beta), I don't expect the same results.20:48
@sonney2kiglesiasg, if you figure that out the only remaining test is evaluation_clustering_simple20:48
@iglesiasgarrrgh20:49
@sonney2ktoday was a tough day in berlin - walking on ice20:49
@iglesiasgI am starting to get concerned that there is something wrong going on here20:49
thoralfsonney2k: I think the only question is if my patch is correct (beside the memory integrity).20:49
@sonney2kI've never seen more people fall...20:49
thoralfsonney2k: But you're idea is good: We could try to solve the same problem and then compare duality gap or something like that.20:52
@sonney2kthoralf, actually we have some test even20:53
@sonney2kjust since yesterday :)20:53
thoralfOr use the weight vector and use the (training) data to compute sth. like average square mean between predicted and trained value20:54
@sonney2kthoralf, no no - it should be 100% identical20:54
thoralfsonney2k: I think the implementation is wrong!20:55
thoralfThe beta is initialized wrong!20:55
thoralfSo why should it return the *same* results?20:55
@sonney2kthoralf, muahh then you should email vojtech / CC michael with your explanation. they are $DEITY20:56
lisitsyn1don't fix it20:56
lisitsyn1:D20:57
lisitsyn1it is going to be a hydra20:57
-!- charry [~charry@91.210.101.37] has joined #shogun20:57
@sonney2klisitsyn, yes marvin :)20:58
thoralflisitsyn1: I already fixed it. ;)20:58
thoralfsonney2k: vojtech-who?20:58
@sonney2kthoralf, vojtech franc20:59
thoralfHmm, okay.21:07
@iglesiasgsonney2k, I screwed up21:10
@iglesiasga revision that yesterday failed, now it doesn't fail21:11
@iglesiasgonly reason I found, because I am compiling using ccache and something is messing around with this21:11
@sonney2kiglesiasg, :`-(21:12
@iglesiasgyep, exactly21:12
@sonney2kiglesiasg, did you uninstall previous releases?21:12
@iglesiasgsonney2k, no21:12
@sonney2kperhaps you have a include conflict?21:13
@sonney2kiglesiasg, ohh you have to!21:13
@iglesiasg:(21:13
@iglesiasghow do I do that?21:13
@iglesiasgI was basically21:13
@iglesiasggit co some_revision21:13
@iglesiasgccache make -j221:13
@iglesiasgsudo make install21:13
@iglesiasgin tests/integration/python_modular $ python tester.py linear_time_mmd.py21:13
-!- negativ [~negativ@91.210.101.170] has joined #shogun21:14
@iglesiasgsonney2k, what should I have done?21:14
@iglesiasgsonney2k, rm -rf build each time? uninstall how?21:14
@sonney2kiglesiasg, always remove the installed shogun dir incl headers/libshogun/python stuff21:14
@sonney2kyes21:14
@sonney2kand rm-rf build too all the time21:14
@wikinginclude conflict should have been fixed already by one of besser82's commit21:15
-!- charry [~charry@91.210.101.37] has quit [Ping timeout: 252 seconds]21:15
@sonney2kwiking, doesn't help if we git bisect old releases21:15
@wikingadding make uninstall target to cmake is a fairly easy process, there are howtos for it so if somebody is up for that should do it21:15
@wikingsonney2k: indeed21:16
@iglesiasgaargh fuck me man, this is so annoying21:16
@sonney2kand that is what we are doing21:16
@wikingsonney2k: bisect within a docker image21:16
@sonney2kwiking, how does that help?21:16
@sonney2kI mean rm'ing is far easier21:16
@wikingwell you can just delete the container...21:16
@wikingand start from the next commit21:16
@iglesiasgsonney2k, btw I am not sure what dirs yo umean with "remove the installed shogun dir incl headers/libshogun/python stuff"21:17
@wikingiglesiasg: e.g. /usr/local/include/shogun /usr/local/lib/libshogun* /usr/local/lib/python2.7/sites-package/shogun21:18
@wikingsomething like this21:18
@wikingif u have installed it21:18
@wikingof cours21:18
@iglesiasgwiking, got it, thanks!21:18
@iglesiasgwiking, I was not sure if there were those ones or something else inside shogun tree dir21:18
@wikingiglesiasg: but you dont have to make install to be able to run ctests21:18
@iglesiasgwiking, the problem is that in this case I cannot try it with ctest21:19
@iglesiasgwiking, I have to run by hand the tester21:19
@iglesiasgwiking, let me show you why21:19
@wikingiglesiasg: then just set PYTHONPATH21:19
@wikingand LD_LIBRARY_PATH21:19
@iglesiasgwhy so?21:19
@iglesiasgto what?21:19
@wikingso that u dont need the make install21:19
@wikingwell why else would you kill yourself with make install? :)21:19
@sonney2ktrue - you don't need make install then21:19
@wikingand then do rm -rf all the time after a check21:20
@iglesiasgdon't get it21:20
@iglesiasgI thought swig interfaces required the sudo make install21:20
@wikingiglesiasg: well u dont need to do make install in order to be able to do a simple check21:20
@wikingiglesiasg: no they dont21:20
@iglesiasgI guess the pythonpath ld_library_path thing fix that21:20
@wikingiglesiasg: what's the script u r trying to run?21:20
@iglesiasgwiking, I am running the integration test on linear_time_mmd.py21:21
@wikingmmm21:21
@iglesiasgso tests/integration/python_modular/tester.py linear_time_mmd.py21:21
@wikingwell you might have a smal problem21:21
@wikingwith the version21:21
@wikingwhere imports are not from modshogun21:21
@wikingbut from shogun21:21
@iglesiasgok...21:21
@iglesiasgI think I am cool with doing install and removing the directories actually21:22
@iglesiasgjust would like to avoid repeating the process all over again :)21:22
@iglesiasgsonney2k, but I think it is better if I try to do this connecting remotely to fatbot21:24
@iglesiasgeach installation takes far too long locally21:24
@iglesiasghmm but will I have problems there with the installation issue??21:25
@sonney2kiglesiasg, don't install21:25
@sonney2klike wiking said21:25
@sonney2kit really is better21:25
@iglesiasgsonney2k, sure ok21:25
@iglesiasgbut tell me what to do instead21:25
@iglesiasgset PYTHONPATH, LD_LIBRARY_PATH21:25
@iglesiasgto?21:25
@iglesiasgI don't really follow why that fixes it21:26
@iglesiasgI would understand that setting them tells make where to install the stuff21:26
@sonney2kiglesiasg, PYTHONPATH=path/to/modshogun ( the directory)21:26
@sonney2kiglesiasg, LD_LIBRARY_PATH=/path/to/libshogun (the dir again)21:26
@sonney2kand then python example.py21:26
@sonney2kiglesiasg, well if there is no default install of shogun -> no clash21:27
@iglesiasgsonney2k, the directory means from where I want to execute the python example.py?21:27
-!- gsomix [~gsomix@46.20.65.64] has joined #shogun21:28
@sonney2kiglesiasg, no the dir where the modshogun.so / libshogun*.so* are in21:28
@iglesiasgsonney2k, i am not sure what to do really21:30
@sonney2kiglesiasg, 1) remove shogun install'd files21:32
@sonney2k2) build again21:33
@sonney2k3) set python/ld_lib path21:33
@sonney2k4) run the example21:33
@sonney2kif any of this fails ask here21:33
gsomixsonney2k, hey. sry, I cannot to participate in today's hacking night. :( my laptop got wet by milk, need to dry.21:33
@iglesiasgaaaah I think I got it now21:33
@iglesiasgsonney2k, the key was telling me that 3) goes after 2) :)21:34
gsomixnothing serious, but keyboard does not work properly21:34
@sonney2kgsomix, you can type pretty well still ;)21:35
gsomixsonney2k, nope21:38
gsomixshift, ctrl, alt keys are crazy21:38
@iglesiasgsonney2k, was it ssh to bug.7nn.de for fatbot?21:41
@sonney2ksonney2k, fatbot.shogun-*21:42
@iglesiasgssh to fatbot.shogun.toolbox.org is not working at all21:42
-!- infomix [~infomix@91.210.100.173] has joined #shogun21:42
@sonney2kiglesiasg, works totally fine here21:43
@iglesiasgit sort of hangs up here21:43
@sonney2kiglesiasg, well then do it locally...21:43
@iglesiasgsonney2k, a little bit weird that I ping without a trouble though21:44
@sonney2klisitsyn, wiking can you still ssh fatbot?21:47
-!- gsomix [~gsomix@46.20.65.64] has quit [Ping timeout: 272 seconds]21:47
@wikingsonney2k: i could about 4 hours ago21:48
@wikingand still i can21:49
@wikingy?21:49
@wiking15388 sonne     20   0 8135m  36m  392 R  99.6  0.1   7685:48 python21:49
@wiking15391 sonne     20   0 12.9g  36m  392 R  99.6  0.1   7726:18 python21:49
@wikingthey are running now for a while21:49
@iglesiasgwiking, would you mind adding a moment my public key?21:51
@iglesiasgwiking, the one that is there is an old one21:51
@wikingiglesiasg: in your home?21:51
@wikingssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDq1H9qVAPm7JZkhy+Mk2cwwExAJJgbnI0S9bT4ZO03caNWuBaP92c/mHNQi7x1+IyCCwd4jqInMrQ6MDTqz1YGZqsWknRzYxT0hs8Z1SSD06gs5APBfkaVpMgyuwLFyKTjqmib9H9vyZnTMj/d+ZYXxlb8DNT78aJqEnDRe8rNps4gCGOq5bJNkcaIV/aL5IJmwLmofjvhtD2O4I6U8pUgat6ulsJFlDTDCha3lEfMcP4KeqH7uSK/pCuT8Z1Xp0LaGgQjOlrHtTRJ73b3U5bXRKPMn+UmifdqXjIB5Vhbj0KDxnqcP3k8cJK27gUDXa8ApTj9oWOMFK6/qQxE/qLB fernando.iglesiasg@gmail.com21:52
@wikingthis is what you have in authorized_keys atm21:52
@wikingif u want something else21:52
@wikinggimme21:52
@wikingand tell me if i should append or overwrite21:52
@iglesiasgwiking, let me see one second21:53
@wikingiglesiasg: u can actually give out public key in public.... that's the idea :P21:57
@wikingbut ok adding21:57
@wikingiglesiasg: tyr now21:57
@wikingshould be working21:57
@iglesiasgwiking, nope actually :S22:00
@wikingmmm interesting22:01
@wikingoh maaan22:02
@wikingyour problem is the username. u r trying with iglesiasg but your username on fatbot is iglesias22:02
@wikingwithout the last g on the end22:02
@wikingiglesias22:03
@iglesiasgwiking,  I tried iglesias too ;)22:03
@iglesiasgssh iglesias@fatbot.shogun-toolbox.7nn.de22:03
@iglesiasgPermission denied (publickey).22:03
@wikingplz try now again22:03
@wikingand and u r trying the wrong host as well22:03
@wiking:DDD22:03
@wikingfatbot.shogun-toolbox.org22:03
@wikingso22:04
@wikingssh iglesias@fatbot.shogun-toolbox.org22:04
@wikingu have 2 more seconds to try as after that i'm gone22:06
@wikingiglesiasg: if u want access to 7nn.de22:06
@wikingi can add there your public key as well22:06
@wikingno actually there i cannot22:06
@wikingas there i dont have sudo22:06
-!- thoralf [~thoralf@91-65-136-3-dynip.superkabel.de] has quit [Quit: Konversation terminated!]22:07
@iglesiasgwiking, .org worked, thanks!22:07
-!- votjakovr [~votjakovr@188.134.46.30] has quit [Quit: WeeChat 0.4.0]22:07
@sonney2kohh man22:07
@sonney2kiglesiasg, now finally figure it out :)22:08
-!- infomix [~infomix@91.210.100.173] has quit [K-Lined]22:18
-!- negativ [~negativ@91.210.101.170] has quit [K-Lined]22:18
-!- rymrym [~rymrym@193.104.110.37] has joined #shogun22:21
@sonney2kargh22:26
-!- pickle27 [d8bfe7da@gateway/web/freenode/ip.216.191.231.218] has quit [Ping timeout: 272 seconds]22:35
-!- shogun-notifier- [~irker@7nn.de] has joined #shogun22:37
shogun-notifier-shogun-data: Soeren Sonnenburg :master * 6b289bd / testsuite/tests/ (2 files): https://github.com/shogun-toolbox/shogun-data/commit/6b289bd5e22d88ec6ad5918063906267a7b65d8122:37
shogun-notifier-shogun-data: update data for fixed clustering evalution22:37
shogun-notifier-shogun: Soeren Sonnenburg :develop * 2483b5a / / (3 files): https://github.com/shogun-toolbox/shogun/commit/2483b5ab18f9195b104e682cfbbb4d49baacd39d22:37
shogun-notifier-shogun: fix ClusteringMutualInformation - init matrix as zero fixing valgrind errors and tests22:37
@sonney2kfinally - iglesiasg so now only the mmd stuff remains!22:38
-!- pickle27 [d8bfe7da@gateway/web/freenode/ip.216.191.231.218] has joined #shogun22:38
@iglesiasgsonney2k, I am doing my best!22:40
@iglesiasgsonney2k, the process is much faster now compiling in fatbot22:42
@iglesiasgsonney2k, I hope it will work out this time to detect the revision that broke the test22:42
besser82sonney2k:  How do we want to get that py_mod SGObj thing started?22:43
@sonney2kbesser82, no idea22:43
@sonney2kbesser82, maybe gsoc task22:44
besser82sonney2k:  and Who to mentor?22:44
@sonney2ktoo big I am afraid22:44
besser82sonney2k:  what about that cmake-stuff && gsoc?22:44
besser82sonney2k:  shall I put that to iglesia's gdrive?22:44
@sonney2kbesser82, sure!22:48
besser82sonney2k:  kk22:48
@sonney2kbesser82, I shouldn't be mentoring more than 2 things so not sure if examples is a good thing for me - though I have a plan... the only problem to decide is whether we want some intermediate language to write the examples in or not22:49
besser82sonney2k:  not really,  keeping the examples lang by lang would be good22:53
besser82sonney2k:  just sharing the reference data22:53
@sonney2kbesser82, no no22:56
@sonney2kit is too much effort to write the examples lang by lang22:56
@sonney2kand they should be very simple just for reference22:56
@sonney2kmore complex stuff should be a notebook22:56
-!- rymrym [~rymrym@193.104.110.37] has quit [K-Lined]22:57
besser82sonney2k:  ok, ok23:00
besser82sonney2k:  but what "intermediate" language to chose?23:01
@sonney2kno idea23:01
lisitsyn1javascript!23:01
@sonney2kmaybe wiking has an idea?23:02
lisitsyn1w00t you are toalking abouuut?23:03
@sonney2klisitsyn, writing shogun examples in some intermediate language / or jinja templates so that it can easily be converted to all $LANG we support23:04
lisitsyn1sonney2k: think that's possible?23:05
lisitsyn1sounds ok but I have no idea23:05
lisitsyn1sonney2k: then we can follow common naming schemes!23:05
lisitsyn1e.g. it is just wrong to have get_stuff in java23:06
lisitsyn1nobody uses that convention in java23:06
lisitsyn1code looks un-idiomatic23:06
lisitsyn1wrrrrong23:06
@sonney2klisitsyn, sure - if you look at the examples they all look the same except for some very minor deviations23:06
@sonney2klisitsyn, yeah swig supports that even...23:06
lisitsyn1hmm then we should have it23:07
lisitsyn1it must be really confusing to use _ in java23:07
@sonney2klisitsyn, yeah ... doing examples just once would have some huge advantage too23:09
pickle27I could be okay to mentor that project if besser82 has enough on his plate23:09
@sonney2klisitsyn, we could return only shogun objects23:09
@sonney2klisitsyn, and then these can be serialized with SerializableAsciiFile23:10
@sonney2kand read back in and compared with equals23:10
@sonney2kso we have cross language tests23:10
@sonney2kand all examples do the same  / it is ensured that they all give the same correct output23:10
@sonney2kpickle27, didn't you have sth else already?23:11
shogun-buildbotbuild #37 of deb4 - python3 is complete: Failure [failed test python modular]  Build details are at http://buildbot.shogun-toolbox.org/builders/deb4%20-%20python3/builds/37  blamelist: Soeren Sonnenburg <sonne@debian.org>23:11
pickle27I have the opencv integration23:11
pickle27I only want to mentor one project but if no one picks that one then i'd be happy to help on examples23:11
@sonney2kpickle27, OK23:13
@sonney2kpickle27, any idea what intermediate language one could use?23:13
@sonney2kI would want to keep it as simple as possible23:13
@sonney2kno idea if jinja2 templates would do...23:14
pickle27honestly most of the examples are already super basic because of the load helper methods23:15
@sonney2kpickle27, yes they are intended to be only reference examples23:15
pickle27I would probably do a second take and move all of the data generation into a contained class23:15
@sonney2kpickle27, we would want to make things even more simple / reference like23:15
@sonney2kpickle27, and put all the higher level stuff into notebooks etc23:15
@sonney2kpickle27, I think jinja2 is sufficient - I mean the shogun stuff has the very same syntax across languages23:17
@sonney2kand data is supposed to be loaded from disk likely even with shogun.23:17
pickle27yeah I think that would work23:17
@sonney2kpickle27, me might want to have a few examples still using this loader crap23:17
@sonney2kbut that again has the same syntax23:17
@sonney2kpickle27, problem is if we pose this for a student we should try *one* minimal example - like e.g. compute the Gaussian Kernel or so23:18
pickle27I would make all the examples use the loader23:18
pickle27then it would be super simple to generate them23:19
@sonney2kpickle27, why not - I think we could have some cmake rule that converts *.py.jinja2 into *.py23:20
@sonney2kand if other *.py's exist it would just not touch things23:20
@sonney2kso we *could* have additional examples if stuff doesn't work out23:20
pickle27yeah totally23:22
pickle27btw just to be clear I'm talking about making sure all these are consistent and then use them to gen examples: https://github.com/shogun-toolbox/shogun/blob/develop/examples/undocumented/python_modular/tools/load.py23:22
@sonney2kpickle27, yes23:23
@sonney2kpickle27, we are still talking about hundreds of examples23:23
pickle27okay just checking we're on the same page23:23
@sonney2kso it will be quite some pile of work23:24
pickle27yeah i know23:24
pickle27there is probably lots that we don't really need slash are basically repeated from unit tests23:24
@sonney2kalso they need to be changed to just return shogun objects - e.g. the kernel's can only be reasonably tested when they return the kernel matrix23:25
@sonney2kbut that would be an SGMatrix - so we need to put them in some CustomKernel or have some other datatype for that that has a .equals23:26
pickle27actually just thinking now, it might be cool to try and move a lot of those really simple tests to be automatically copied from unit tests23:27
pickle27cause we're packing a lot of duplication right now aren't we?23:27
pickle27I know that all of the examples I wrote for gsoc were basically the same as my unit tests23:28
pickle27we could have a flag or something that says this unit test is also a basic example and then it gets copied in23:28
pickle27anyways I got to run but I might look into that after Im done on the website see if I can put something together23:29
-!- pickle27 [d8bfe7da@gateway/web/freenode/ip.216.191.231.218] has quit [Quit: Page closed]23:30
@sonney2kpickle27 it is actually even more simple - a unit test would compute e.g. the gaussian kernel and check if values are as expected - in the example you just have the GaussianKernel and done23:31
shogun-buildbotbuild #2142 of deb3 - modular_interfaces is complete: Failure [failed test python modular]  Build details are at http://buildbot.shogun-toolbox.org/builders/deb3%20-%20modular_interfaces/builds/2142  blamelist: Soeren Sonnenburg <sonne@debian.org>23:47
--- Log closed Tue Jan 21 00:00:01 2014