Open in new window / Try shogun cloud
--- Log opened Tue Nov 26 00:00:40 2013
-!- shogun-notifier- [] has quit [Quit: transmission timeout]00:18
@iglesiasggood night guys00:53
-!- iglesiasg [] has quit [Quit: Ex-Chat]00:53
-!- new_lido_ is now known as new_lido_afk02:27
-!- new_lido_afk is now known as new_lido_02:31
-!- sonne|osx_ [] has joined #shogun03:02
-!- sonne|osx [] has quit [Ping timeout: 272 seconds]03:05
-!- sonne|osx_ is now known as sonne|osx03:05
shogun-buildbot_build #533 of nightly_all is complete: Failure [failed test]  Build details are at
-!- new_lido_ [~walid@] has quit [Ping timeout: 245 seconds]03:41
-!- new_lido [~walid@] has quit [Ping timeout: 272 seconds]03:42
-!- new_lido [~walid@] has joined #shogun03:53
-!- Boeke [] has joined #shogun04:27
Boekehey can any one tell me the best way to open a hdf5 file with shogun is there still a direct method?04:28
Boekelike HDF5File04:28
-!- new_lido [~walid@] has quit [Ping timeout: 264 seconds]04:42
BoekeI suppose it is also important to say that I am using the python modular interface.04:44
shogun-buildbot_build #629 of nightly_default is complete: Success [build successful]  Build details are at
-!- Boeke [] has quit [Quit: Leaving]07:18
-!- sonne|osx [] has quit [Quit: sonne|osx]07:35
-!- sonne|osx [~sonne@] has joined #shogun08:22
sonne|osxshogun-buildbot_: force build --branch=develop nightly_all08:22
shogun-buildbot_build forced [ETA 10m57s]08:22
shogun-buildbot_I'll give a shout when the build finishes08:22
shogun-buildbot_build #534 of nightly_all is complete: Success [build successful]  Build details are at
-!- besser82|laptop [~besser82@fedora/besser82] has joined #shogun08:33
-!- besser82 is now known as Guest3103008:34
-!- besser82|laptop is now known as besser8208:34
sonne|osxhey besser82 !08:36
-!- Guest31030 [~besser82@fedora/besser82] has quit [Ping timeout: 252 seconds]08:36
sonne|osxlong time no see08:36
-!- shogun-notifier- [] has joined #shogun08:36
shogun-notifier-shogun: Soeren Sonnenburg :develop * 48815d9 / src/interfaces/modular/SGBase.i:
shogun-notifier-shogun: for java / c# add dummies for serialization for  SGRefObject08:36
besser82sonne|osx: Yes, those business duties you know?!?  :D08:37
sonne|osxbesser82: I managed to get protobuf to work in the meantime so you can do sth else or do it besser82 :)08:37
besser82sonne|osx: I'm going to push your requested cmake changes today  ;)08:37
sonne|osxbetter I mean ;)08:37
besser82sonne|osx: like possibly improving protobuf && enabling build of modules from pre-compiled / existing libshogun.so08:38
sonne|osxexternal would be graet08:39
sonne|osxthen i could get the debs towork08:39
* sonne|osx off train08:40
-!- sonne|osx [~sonne@] has quit [Quit: sonne|osx]08:40
besser82sonne|work: btw. is testsuite failing on 32-Bits arches for you, too?08:42
besser82sonne|work: on Fedora && RHEL ~10 test are repeatedly failing with ppc / ix86 / armv7h08:43
besser82sonne|work: x86_64 (amd64) and ppc64 are running fine...08:43
shogun-buildbot_build #1423 of rpm1 - libshogun is complete: Failure [failed test]  Build details are at  blamelist: Soeren Sonnenburg <>09:01
-!- new_lido [~walid@] has joined #shogun09:05
shogun-buildbot_build #358 of FC19 - libshogun is complete: Success [build successful]  Build details are at
shogun-buildbot_build #352 of FCRH - libshogun is complete: Success [build successful]  Build details are at
sonne|workbesser82: back09:24
sonne|workbesser82: which ones?09:24
besser82sonne|work: sry, been afk...  just a momemt09:28
shogun-buildbot_build #2057 of deb3 - modular_interfaces is complete: Failure [failed examples and unit tests]  Build details are at  blamelist: Soeren Sonnenburg <>09:28
besser82sonne|work:  <---  reproducible on all 32-Bit arches, but HDF5 which fails on 64-Bits, too...09:31
besser82sonne|work: does hdf5 need inet-connectivity?09:32
besser82sonne|work: the test I mean...09:32
sonne|workbesser82: the mldata one yes - so disable it - it is downloading some data set from
besser82sonne|work: kk, will do so  ;)  thx09:33
-!- thoralf [] has joined #shogun09:33
sonne|workbesser82: the others seem to be numerical differences like 1E-15 evaluates to 1.0000000000000001e-15.09:34
besser82sonne|work: like precision-difference for 64 vs. 32 bits?09:36
besser82sonne|work: or related to other problems?09:36
sonne|workbesser82: yeah I guess so but I better investigate further. I need to run this on some legacy machine to be sure - the differences were really minor but still09:37
besser82sonne|work: I can reproduce that in x86 VM and chroot...09:38
besser82sonne|work: bare metal (Pentium4), too...09:39
sonne|workthoralf: hey good morning! thanks for the patch! do you recall if iglesiasg agreed that we don't need serialization for output?09:39
thoralfsonne|work: The problem with m_output also applies to StructuredLabels::m_labels :(09:41
thoralfsonne|work: See my commend.09:41
sonne|workdon't get shogun mails at work...09:41
* sonne|work checks the PR09:42
sonne|workthoralf: nargh09:42
thoralfDidn't see that in advance, sorry.09:43
sonne|workthoralf: this would rather mean we need some 'cheap' serialization too09:43
sonne|workthoralf: not your fault - fernando didn't see it coming either09:44
thoralfsonne|work: Okay, SG_ADD is for serialization, right?  So why wo we have to initialize this at object creation?  Maybe just call all these SG_ADDs only when serializing?09:45
sonne|workthoralf: in principle yes but we need this for deep_copy of objects / equals too09:45
thoralfI see.09:46
thoralfMaybe introducing a SG_ADD_UNSERIALIZABLE(...) to add arbitrary variables?09:46
thoralfWould be a hack, since it moves the problem to runtime...09:47
-!- new_lido [~walid@] has quit [Read error: Connection reset by peer]09:48
sonne|workthoralf: we have the other overhead like the io / parallel etc objects09:53
sonne|workthoralf: but I agree it would be nice to say have some extra class that you just plug in09:54
sonne|workand then you get serialization09:55
-!- iglesiasg [] has joined #shogun09:55
-!- mode/#shogun [+o iglesiasg] by ChanServ09:55
sonne|workthoralf: it really was never meant to be efficient in that respect09:56
sonne|workthoralf: the assumption was that any SGObject is huge anyway09:56
sonne|workas in holds a huge matrix or vector or sth09:56
sonne|workwhile it is true we could reduce SGObject memory footprint a bit09:56
sonne|workI just don't think that the RealNumber approach for more than debugging is the right way(tm)09:57
sonne|workyou should rather have some aggregated structure09:57
sonne|worksay RealNumbers or RealVector(s)09:57
sonne|workand then have stuff working on this09:57
sonne|workiglesiasg: hey!09:58
sonne|workiglesiasg: seen thoralf's comments?09:58
@iglesiasghey there09:58
@iglesiasgyeah, I am following the conversation09:58
thoralfHey iglesiasg - didn't see you sneaking in ;)09:58
sonne|worklisitsyn: any ideas why is happening09:58
sonne|worklisitsyn: location: class org.shogun.SGRefObject09:59
sonne|work                this.save_serializable(tmpFile);09:59
sonne|worklisitsyn: I did add SERIALIZABLE_DUMMY(shogun::SGRefObject);09:59
sonne|worklisitsyn: but it is not used?! any ideas?09:59
thoralfsonne|work: Well, maybe we simply could have subclasses of StructuredLabels for each label type.  As labels usually don't live alone.10:00
sonne|workthoralf: sounds better - iglesiasg?10:00
sonne|workiglesiasg: any thoughts?10:00
@iglesiasgI understand this is how it is done right now, isn't it?10:01
@iglesiasgthese subclasses would be the current Sequence, RealNumber, FactorGraph, etc10:01
@iglesiasgwell, without etc, I think there are no more than those :D10:01
thoralfiglesiasg: No.  Dropping these "Data" classes and putting them into the "Labels" classes.10:01
thoralfDrop RealNumber and create RealSOLabels10:02
thoralfextending StructuredLabels10:02
thoralfBecause objects (in shogun) are too expensive to have millions of them.10:02
@iglesiasgthere is a MulticlassSOLabels, SequenceLabels and so that extend StructuredLabels10:02
thoralfiglesiasg: But they internally create an object for every label.10:03
-!- zxtx [] has quit [Ping timeout: 264 seconds]10:03
@iglesiasgthoralf,  so how would they look like now?10:03
-!- sonne|work [] has left #shogun []10:04
-!- sonne|work [] has joined #shogun10:04
thoralfI'm just wondering if it would be possible (with finite effort) to refactor the SO labels... Maybe by just making StructuredLabels an interface and each Labels class implementing it's own (efficient) structures.10:05
thoralfe.g. RealSoLabels only dealing with SGVector<float> internally.10:05
@iglesiasgok, I think it should be possible10:07
@iglesiasghowever, I think that the type of each label should still have the same superclass10:07
@iglesiasgthe current StructuredData10:07
thoralfEhrm.  I was just trying to avoid StructuredData. ;)10:09
-!- zxtx [] has joined #shogun10:11
-!- naywhayare [] has quit [Ping timeout: 260 seconds]10:12
@iglesiasghehe ok10:13
thoralfsonne|work: Btw., why did we make SGDynamicRefObjectArray a SGRefObject instead of CSGObject?10:13
sonne|workiglesiasg: please think this through thoroughly - we don't want to waste time here10:13
thoralfsonne|work: SGDynamicRefObjectArray could by a CSGObject10:14
@iglesiasgsonne|work, sure10:14
sonne|workthoralf: well it didn't support serialization anyway10:14
@iglesiasgthoralf, but I don't think I see the problem we are trying too solve. Last week we talked about the overhead introduced because StructuredData extended CSGObject, fine.10:15
@iglesiasgthoralf, if StructuredData does not extend it, what is the new problem such that you would like to get rid of StructuredData?10:15
thoralfThat we lose serialization/equals/copy on StructuredLabels, because it's StructuredData entries do not support these operations.10:16
@iglesiasgGot it then10:17
thoralfI think we could either abort the refactoring here or push it further.10:18
@iglesiasgI am trying to think how would the SO framework look like if there was no StructuredData base class10:19
thoralfDepends on how easy it would be to do solve it.10:19
@iglesiasgthoralf, how would for instance the argmax be defined in the CStructuredModel?10:21
@iglesiasgIn case there is no StructuredData base class.10:21
@iglesiasgThe function signature.10:22
thoralfGood point.10:22
@iglesiasgA completely different approach10:24
@iglesiasgWe not separate CSGObject?10:24
thoralfParse error ;)10:24
@iglesiasgI believe that the part that gives the overhead is mainly due to model selection and cross validation stuff10:24
-!- naywhayare [] has joined #shogun10:25
thoralfiglesiasg: Yes.10:28
thoralfWe have four different parameter objects in CSGObject: m_parameters, m_model_selection_parameters, m_gradient_parameters, m_parameter_map10:29
@iglesiasgmaybe these guys should be moved to another class then10:29
thoralfparameter/parameter_map are neede for serialization/clone/deep_copy/equals.10:29
@iglesiasgis the overhead too large if only parameter/parameter_map are kept?10:29
thoralfBut at least m_model_selection_parameters and m_gradient_parameters are not obvious.10:29
@iglesiasgIf it is, then this approach has no sense whatsoever.10:30
thoralfWell, the overhead is mainly introduced by these 4 guys.10:30
thoralfSo avoiding two of them would be a good deal ;)10:30
thoralfLooks like they could be extracted easily from CSGObject.10:33
@iglesiasgI suggest to send a mail to Heiko in any case, I think he did that so may point out something to take into account10:39
thoralfOf course.10:42
thoralfBut I think this is getting too complicated to be fixed easily.10:42
thoralfsonne|work: Agree that we stop working on the issue here?  Too many complications everywhere...10:43
thoralfToo many dependencies.10:44
thoralfAnd iglesiasg, thanks for your time.10:44
-!- naywhayare [] has quit [Ping timeout: 264 seconds]10:46
-!- naywhayare [] has joined #shogun10:54
-!- thoralf [] has quit [Quit: Konversation terminated!]10:58
-!- thoralf [] has joined #shogun10:59
@iglesiasgthoralf, no problem at all :)11:02
sonne|workiglesiasg: the right fix is not trying to squeeze every byte out of sgobject - it was never intended to be a cheap object.11:03
-!- thoralf [] has quit [Client Quit]11:03
sonne|workiglesiasg: I mean there is a reason why we have a CLabels class and not one CLabel object and then an array of them11:04
-!- thoralf [] has joined #shogun11:04
sonne|workiglesiasg: so please think about how we can use aggregated structures and return e.g. multiple things in argmax11:05
-!- thoralf [] has quit [Client Quit]11:08
-!- thoralf [] has joined #shogun11:09
-!- thoralf [] has quit [Client Quit]11:13
-!- thoralf [] has joined #shogun11:14
-!- thoralf [] has quit [Client Quit]11:18
-!- thoralf [] has joined #shogun11:19
-!- thoralf [] has quit [Client Quit]11:24
-!- thoralf [] has joined #shogun11:24
-!- thoralf [] has quit [Client Quit]11:29
-!- thoralf [] has joined #shogun11:29
-!- thoralf [] has quit [Client Quit]11:34
-!- thoralf [] has joined #shogun11:34
-!- shogun-notifier- [] has quit [Quit: transmission timeout]11:36
-!- zxtx [] has quit [Ping timeout: 245 seconds]11:37
-!- thoralf [] has quit [Client Quit]11:39
-!- thoralf [] has joined #shogun11:39
-!- thoralf [] has quit [Client Quit]11:44
-!- thoralf [] has joined #shogun11:44
-!- thoralf [] has quit [Client Quit]11:49
-!- thoralf [] has joined #shogun11:49
-!- thoralf [] has quit [Client Quit]11:54
-!- thoralf [] has joined #shogun11:54
-!- thoralf [] has quit [Ping timeout: 246 seconds]12:00
-!- iglesiasg [] has quit [Quit: Ex-Chat]12:52
-!- besser82 [~besser82@fedora/besser82] has quit [Ping timeout: 252 seconds]13:03
-!- besser82 [] has joined #shogun13:43
-!- besser82 [] has quit [Changing host]13:43
-!- besser82 [~besser82@fedora/besser82] has joined #shogun13:43
-!- lisitsyn [~lisitsyn@] has quit [Ping timeout: 248 seconds]13:46
-!- zxtx [] has joined #shogun13:47
-!- thoralf [] has joined #shogun13:56
-!- zxtx [] has quit [Ping timeout: 272 seconds]14:01
thoralfsonne|work: Hmm.  Iglesias said (void*), that's true, but suppose we're having an array of (void*) in StructuredLabels, how would we compare these?14:01
-!- zxtx [] has joined #shogun14:13
sonne|workthoralf: I didn't yet understand what is wrong with a typed data container...14:17
thoralfsonne|work: Well, as soon we put something into StructuredLabels, which is *not* CSGObject, we'll have this serialization problem.  Right?14:21
sonne|workthoralf: well SGVector/SGMatrix/Sparse*/String* etc works of course14:21
thoralfOut of the box or with custom code?14:22
sonne|workthoralf: out of the box14:22
thoralfsonne|work: How does it work?14:22
-!- lisitsyn [] has joined #shogun14:23
thoralfNo SG_ADD methods in SGVector.14:23
sonne|workthoralf: SG_ADD(&my_vector, "bla", "bla", MS_NOT_AVAILABLE);14:23
sonne|workSG_ADD(&my_matrix, "bla", "bla", MS_NOT_AVAILABLE);14:23
thoralfWell, in this case we could simply write "SG_ADD((CSGObject**)&m_outputs, "outputs", "Structured outputs", MS_NOT_AVAILABLE);" in StructuredLabels/MAPInference and we're done?14:25
thoralfWithout casting it to CSGObject**?14:25
sonne|workthoralf: if m_outputs is SGVector or so yes14:26
thoralfIt is not.14:26
sonne|workthoralf: what is it?14:29
thoralfm_output is no SGVector14:29
sonne|workthoralf: but what then?14:29
thoralfI'm trying to figure out what's neccessary to get this SG_ADD back on m_outputs/m_labels... e.g. DynamicRefObjectArray* or StructuredData*14:32
thoralfI still don't understand this magic.14:32
sonne|workthoralf: which class is this you are talking about - I haven't used the SO framework nor am I the architect so no idea14:33
sonne|workthoralf: in any case there is no magiv14:33
sonne|workthoralf: you just SG_ADD() variables you want to register14:33
thoralfCan these variables be of any type?14:34
lisitsynany of supported ;)14:34
sonne|workthoralf: and all you can do is add CSGObject data types and the SG{Vector,Matrix,SparseMatrix} types and scalars like bool / int32_t etc14:34
sonne|worklisitsyn: ahh you there - seen my question above?14:35
lisitsynsonne|work: no, let me read llogs14:35
lisitsynsonne|work: uhm what question sorry?14:35
sonne|work(09:59:04 AM) sonne|work: lisitsyn: any ideas why is happening14:36
sonne|work(09:59:16 AM) sonne|work: lisitsyn: location: class org.shogun.SGRefObject14:36
sonne|work(09:59:16 AM) sonne|work:                 this.save_serializable(tmpFile);14:36
sonne|work(09:59:37 AM) sonne|work: lisitsyn: I did add SERIALIZABLE_DUMMY(shogun::SGRefObject);14:36
sonne|work(09:59:45 AM) sonne|work: lisitsyn: but it is not used?! any ideas?14:36
lisitsynhmm interesting14:37
sonne|worklisitsyn: we refactored SGObject so that it derives from SGRefObject which in turn only does the refcounting14:37
lisitsynsonne|work: yeah I was following it a bit14:38
sonne|worklisitsyn: so it makes sense to be able to expose any SGRefObject to the modular interfaces (since we only need ref/unref)14:38
lisitsynsonne|work: quite strange it didn't work14:39
sonne|workthoralf: clear now?14:40
thoralfsonne|work: Well, yes.  But seems unsolvable (within finite time).14:43
lisitsynsonne|work: can't see why it fails14:44
sonne|worklisitsyn: hmmh  I don't see it either14:44
sonne|workthoralf: you know more about the framework than me - what is the current requirement?14:44
lisitsynsonne|work: may be SGBase.i:313?14:45
lisitsynthe class is referenced before extended14:45
lisitsynI don't know if that could cause it14:45
-!- Saurabh7 [~Saurabh7@] has joined #shogun14:46
sonne|worklisitsyn: could be no idea - try it14:46
-!- Saurabh7 [~Saurabh7@] has quit [Client Quit]14:46
-!- Boeke [] has joined #shogun14:47
sonne|workBoeke: what was your question again?14:47
sonne|workBoeke: ahh yes HDF514:48
sonne|workBoeke: just use HDF5File14:48
@wikinglisitsyn: eigen is fucking awesome14:48
lisitsynwiking: yeah eigen is ok ;)14:49
lisitsynwhat makes you that excited?14:49
Boekeyeah ok. I can't load it I am going to go back and see if I can't do what I did last time I had a problem loafing a module.14:49
thoralfsonne|work: No real requirement?  We tried to refactor StructuredData, but it didn't work out.  ;)14:50
thoralfsonne|work: My requirement was: Use less memory, but that's already solved.  You said, we should refactor StructuredData. ;)14:51
@wikinglisitsyn: well that it has all the modules for sparsesuite as well14:51
@wikinglisitsyn: and cholmod already has support for gpu based factorization14:52
BoekeSo the error I get is vague and different from last time14:53
BoekeImportError: cannot import name HDF5File14:53
lisitsynwiking: viennacl is interesting in that part14:53
@wikinglisitsyn: yeah i've checked yesterday14:54
lisitsynwith good interfacing with eigen14:54
@wikinglisitsyn: it has one factorizer (incomplete cholesky) for sparse matrices14:54
@wikingor no14:55
lisitsynwiking: they have interesting thing with these 'tags'14:56
lisitsynto designate algorithm to be used14:56
sonne|workBoeke: but hdf5 got detected by cmake?14:56
@wikingso actually i'm going to go with having the factorization algo as a template module14:56
@wikinglisitsyn: as if u have viennacl then u might want to do opencl based factorizatio14:57
lisitsynwiking: what do you do and why? ;)14:57
@wikinglisitsyn: well currently a KKT equation solver14:57
@wikinglisitsyn: the idea is that it could be used by other optimizers14:58
lisitsynis it lacked currently?14:58
lisitsynI have no idea about that14:58
@wikingbut i'll use it in my own qp solver14:58
@wikinglisitsyn: well the thing is that htis is always reimplemented eveyrwhere14:58
lisitsynI see14:58
@wikingand of course it's never super optimized14:58
Boekeoh no I realized I needed hdf5 after I installed. I am new to 9 gig data sets.14:59
@wikinglisitsyn: and since matrix factorization is heavily used in this tasks... i thought that it would be good to have the factorization methods as a plugin15:00
BoekeI guess I need to uninstall and reinstall again right?15:01
@wikingBoeke: reinstall should be fine15:01
@wikingas it'll overwite your previous install15:02
Boekeok thanks15:02
Boekecmake is throwing a lot of errors trying to add custom rules to things that already have them15:17
@wikingtry to rm -rf build15:17
@wikingand then create build dir and rerun cmake15:17
@wikingshould help with such problems15:17
BoekeI still encounter that error15:25
-!- sonne|work [] has quit [Read error: Connection reset by peer]15:32
-!- sonne|work1 [] has joined #shogun15:33
@wikingBoeke: can u pastebin that error plz15:45
Boekeok I will try15:46
@wikingBoeke: if u really need shogun out of box just use our docker image15:47
@wikingBoeke: otherwise just copy-paste us somehow your erro15:47
@wikingBoeke: did u use shogun before 3.0?15:50
@wikingok that all makes sense now15:51
@wikingBoeke: so in /home/alex/shogun-3.0.0/src/interfaces/python_modular15:52
@wikingdelete those files that the cmake is complaining about15:52
@wikingi.e. Machine.i Multiclass_includes.i Distance.i15:52
@wikingand rerun in your build dir the cmake15:52
besser82sonne|work, wiking, lisitsyn, thoralf: any objections moving the generation of documented examples from external-script to cmake-internal?  So the documented examples are actually generated out of source-tree?16:00
besser82same for config.h / version.h, too?16:01
thoralfAnd class_list.cpp? ;)16:01
@wikingbesser82: no16:01
@wikingbesser82: go ahead16:01
besser82thoralf: class_list.cpp is autogenerated?16:01
thoralfbesser82: Yes.16:02
@wikingbesser82: the only problem you'll have with the examples that it depends on the data submodule16:02
@wikingbesser82: and it's hardcoded in the examples (the path)16:02
@wikingbesser82: it's like ../data/<datafilename>16:02
@wikingbesser82: so that'll need to be generated somehow within the build dir16:03
thoralfbesser82: Yields to problems some times, if you're building release/debug versions alternately.16:03
@wikingbesser82: yes class_list is generated as well16:03
besser82wiking: setting up a symlink isn't the problem  ;)16:03
@wikingbesser82: yeah i know i'm just saying that it should be in the right 'level' :P16:04
besser82wiking:  ;)16:04
@wikingbesser82: that goes really for all the examples (libshogun, *_modular etc)16:04
-!- zxtx [] has quit [Read error: Operation timed out]16:04
besser82wiking: Seen it  ;)16:04
besser82wiking, thoralf: anything else being autogenerated?16:05
@wikingbesser82: noup... just in unit tests16:05
@wikingbesser82: but those are generated within the build dir16:05
besser82wiking: so actually with cmake magic, rye?16:05
@wikingso that should be alright16:05
besser82wiking: any objections to alter the generation of the modules in a way they will properly use ccache?16:09
besser82wiking: like speeding rebuilds somewhat insane  ;)16:09
sonne|work1besser82: no we like long compile times!16:10
@wikingbesser82: well we use already ccache16:10
@wikingbesser82: both for modular interfaces and libshogun16:10
@wikingi.e. ccache and ccache-swig16:11
@wikingif of course ccache or ccache-swig is available16:11
@wikingso i dont see how that could be still cached...?16:11
@wikingor in another way16:11
besser82wiking: I'm using ccache,too; but builds still take ~10 minutes for me  :(16:12
besser82wiking: so there must be a solution  :-P16:12
@wikingbesser82: well ccache is being prepended for each compilation if aavailable16:12
-!- iglesiasg [~iglesias@2001:6b0:1:1041:e4ea:290f:b6ca:d6e] has joined #shogun16:13
-!- mode/#shogun [+o iglesiasg] by ChanServ16:13
@wikinghence i really dont see how that can be still changed to be faster16:13
@iglesiasgthoralf, they there!16:13
thoralfHey iglesiasg16:13
besser82wiking: but infact a lot of files of the modules are with the same name16:13
besser82wiking: that shreds most of ccache afaik....16:14
@iglesiasgthoralf, I have been thinking about the SO refactoring. Let me ask you16:14
besser82iglesiasg: hi!16:14
@wikingbesser82: yes but the thing is that the swig file for all the interfaces a monolithic file...16:14
thoralfiglesiasg: Go ahead. :)16:14
@wikingbesser82: modshogun.i16:14
@iglesiasgthoralf, if there is no StructuredData class, how will serialization be supported?16:14
@iglesiasgbesser82, sup!16:15
@wikingbesser82: and that is being ccached16:15
thoralfiglesiasg: I've been thinking about that, but I don't know.16:15
@iglesiasgthoralf, ok, I see16:15
thoralfiglesiasg: With my current PR it wouldn't work any more. ;)16:15
@iglesiasgthoralf, because I guess the argmax issue we talked about before could be fixed by using void* somewhere16:16
besser82wiking: let me investigate some optimization possibilities16:16
thoralfiglesiasg: Yeah, but still: StructuredLabels must contain something that can be serialized.16:17
besser82wiking: i possibly may find some....16:17
@iglesiasgthoralf, exactly16:17
thoralfiglesiasg: Either array of CSGObject or SGVector or SGMatrix, whatever16:17
@iglesiasgthoralf, yes. However, SGVector and SGMatrix are just designed to contain primitive data types16:17
@iglesiasgwhich makes sense to me16:17
thoralfiglesiasg: Currently this is unsolved - independently of the argmax16:17
thoralfiglesiasg: I know.16:18
@wikingbesser82: note that we had problems with previously trying to separate swig files into modules.... this is why we have now one monolitic swig file (modshogun.i)16:18
@iglesiasgthoralf, but I still don't understand why can't we have a lightweight CSGObject16:18
sonne|work1iglesiasg: we could but it doesn't fix the problem16:18
besser82wiking: mhh....16:19
@iglesiasgsonne|work1, why not?16:19
thoralfiglesiasg: I already tried - have a look at SGRefObject.16:19
besser82wiking: lemme try...16:19
thoralfiglesiasg: There are many dependencies on every thing implemented in CSGObject.16:19
@wikingbesser82: if we could use swig modules we could use ccache on those... but at last we tried it was not possible, as the modular interfaces were just crashing16:19
sonne|work1iglesiasg: you still have a comparably huge overhead for just storing one float16:20
thoralfiglesiasg: We could outsource gradient/model selection stuff as well, but I think thats all.16:20
sonne|work1iglesiasg: the right fix would be to use an object that contains not just 1 float but say a vector or matrix of floats16:20
thoralfsonne|work1: But SO works differently - it should be able to handle arbitrary outputs.16:20
@iglesiasgsonne|work1, thoralf, however, I still think that the idea of storing a float as a structured label does not really represent what the SO stuff is usef for16:21
thoralfsonne|work1: Trees, sequences, sets, etc.16:21
sonne|work1thoralf: arbitrary outputs composed of basic data types16:21
@iglesiasgwhat I mean is that we should not aim at designing the structured labels so that they are efficient when CStructuredLables is basically an array of floats16:21
thoralfsonne|work1: Of course, but still: How would you represent a class that can hold arbitrary combinations of basic types?16:23
sonne|work1thoralf: how many of these class instances do you expect?16:24
sonne|work1normally I would say CSGObject16:24
sonne|work1butif #instances is too big you need some aggregated labels for efficiency reasons16:25
@iglesiasgthoralf, wiking, you guys have experience with SVMStruct code, don't you?16:25
thoralfiglesiasg: Yes.16:25
@iglesiasgthoralf, how do they solve this issue we are facing here?16:25
@iglesiasgIIRC, in the doc it is claimed that you can pretty much implement your own labels as you want and still use their solvers16:26
thoralfiglesiasg: You can define your own struct to represent your outputs, then compile it. ;)16:26
@iglesiasgbut internally, in the ssvm solver for instance, this struct must be handled somehow16:27
thoralfiglesiasg: And you have to implement all stuff that deals with your data type.16:27
@iglesiasgI don't think I am making my point very clear.... argh16:27
thoralfWe need a whiteboard. ;)16:28
thoralfFeels like a big misunderstanding. ;)16:28
-!- lisitsyn [] has quit [Quit: Leaving.]16:29
@iglesiasgthoralf, let me try to explain16:29
@iglesiasgthoralf, so SVMStruct supports any data type the user may define16:29
@iglesiasgthoralf, agree?16:29
thoralfYes, but it doesn't have any overhead like additional attributes you didn't define.16:30
@iglesiasgthen it must be that the user defined struct does not extend any internal SVMStruct class16:30
@iglesiasgthoralf,  good so far?16:31
@iglesiasgAll right.16:31
thoralfAnd that's the main difference to shogun, btw.16:31
thoralfThe implementation is similar, except that svmstruct does not provide serialization, etc.16:32
@iglesiasgThen, how do the signatures of the functions that use this user defined data look like?16:32
@iglesiasgBecause these functions must be used somewhere in their code for sure16:32
thoralfloss(label *ytrue, label *ypred);16:32
@iglesiasgall right16:33
thoralfiglesiasg: SVMstruct only handles one implemented type at a time.16:33
thoralfYou need another type?  Recompile.16:33
thoralfIt's the C-way.  Not real C++. ;)16:33
@iglesiasgWe cannot afford doing that unfortunately16:34
sonne|work1thoralf: but you surely could do that in C too16:35
thoralfsonne|work1: Is there a way to implement custom comparator/serialization functions for StructuredLabels?16:35
@iglesiasgsonne|work1, do you mean  without the need to recompile?16:35
thoralf(an easy way preferably ;))16:35
sonne|work1iglesiasg: yes16:36
@iglesiasgsonne|work1, I'd say that you can do that in C using void*16:36
sonne|work1iglesiasg: well you can fake classes by handing over structs that contain function pointers...16:37
sonne|work1that is how the linux kernel guys do it16:37
thoralfPseudo-Objective-C.  Loving it. ;)16:37
@iglesiasgthoralf, the only thing I can think about now is to make a new StructuredData base class that supports everything we need (serialization, lightweight, hopefully ref count like the one in SGObject)16:37
@iglesiasgthoralf, but that sounds pretty crazy :D16:37
sonne|work1iglesiasg: but that is not efficient right?16:38
@iglesiasgsonne|work1, efficient in terms of the time/effort if may take?16:38
sonne|work1iglesiasg: efficient memory overhead16:38
@iglesiasgsonne|work1, why not? Ref count would probably include little overhead, but I don't see how serialization would include any overhead16:40
thoralfiglesiasg: The SG_ADD stuff introduces the overhead at object creating.16:40
@iglesiasgthoralf, what does SG_ADD do internally?16:41
@iglesiasgref count stuff, model selection stuff, both?16:41
sonne|work1well we could say we have one SerializationFramework class16:41
sonne|work1and all SGObjects have that as a member16:41
sonne|work1which is null usually16:42
sonne|work1but only when we do a serialization (load/save/comparison)16:42
sonne|work1the object gets created16:42
thoralfsonne|work1: Sounds pretty good.16:42
thoralfLazy initialization.16:42
sonne|work1this certainly makes sense (has some other issue - like you serialize and suddenly it eats memory like hell)16:44
sonne|work1but no idea if this is the right approach - I would rather use some aggregated labels16:45
@iglesiasghow would these aggregated labels look like?16:46
thoralfsonne|work1: The easiest way would be to have skinny StructuredData objects.  Nothing more than structs.16:46
sonne|work1iglesiasg: e.g. not a real number but a vector - for multiple labels16:47
thoralfsonne|work1: So basically structs+refcounting.  Minimal overhead.16:47
sonne|work1thoralf: but we don't have that because we want serialization of ?16:48
thoralfsonne|work1: That brings us back to my above question: Is there a way to implement custom comparator/serialization functions for StructuredLabels? ;)16:49
thoralfStructuredLabels taking care of it's own stuff.  So we can implement StructuredData as skinny as possible.16:50
sonne|work1thoralf: nah sounds more difficult than the optimized SGObject16:51
@iglesiasgCSGObject::equals is virtual so why not16:51
@iglesiasgabout serialization idk16:51
@iglesiasgwhere is the serialization code btw?16:52
sonne|work1iglesiasg: SGObject.cpp and Parameter*16:52
@iglesiasgload_serializable I see16:53
@iglesiasgand save, etc16:53
@iglesiasgall right, I have to run to a lab session now, see you later16:54
-!- iglesiasg [~iglesias@2001:6b0:1:1041:e4ea:290f:b6ca:d6e] has quit [Quit: Ex-Chat]16:55
-!- sonne|osx [~sonne@] has joined #shogun17:19
* sonne|osx sighs17:21
BoekeHi cmake is working but can't find "HDF5_DIR: The directory containing a Cmake configuration file for HDF5."17:24
sonne|osxBoeke: please paste the output of cmake ... do you have hdf5 developer files installed?17:28
-!- shogun-notifier- [] has joined #shogun17:29
shogun-notifier-shogun: Soeren Sonnenburg :develop * 3761d82 / src/interfaces/modular/SGBase.i:
shogun-notifier-shogun: add dummy before doing ref / unref magic17:29
BoekeI do have the dev files installed. Cmake doesn't throw any errors I could generate and exit17:30
BoekeI just don't think that will get HPF5File installed.17:31
@wikingBoeke: can u copy paste the whole output of cmake somewhere17:33
Boekecmake or ccmake ?17:34
sonne|osxBoeke: cmake in the beginning gives you an overview what optional features you have installed17:34
sonne|osxBoeke: please paste that so we know hdf5 is among the detected libs17:34
BoekeI will do that17:34
-!- sonne|osx [~sonne@] has quit [Quit: sonne|osx]17:40
Boekeso it did find HDF517:43
Boekeok ccmake can't find the directory but cmake knows the package is there. If I generate a make file with ccmake will it know I have HDF5?17:47
@wikingu might want to read this:
shogun-buildbot_build #1424 of rpm1 - libshogun is complete: Success [build successful]  Build details are at
Boekeok will do17:52
besser82wiking: Where do I find this `modshogun.i`-swig-file?17:58
@wikingbesser82: src/interfaces/modular17:59
@wikingthere are all the common swig .i files17:59
shogun-buildbot_build #320 of precise - libshogun is complete: Failure [failed test]  Build details are at  blamelist: Soeren Sonnenburg <>18:00
besser82wiking: Thanks! As I suspected  ;)18:00
besser82wiking: And there lies the source of shredding the ccache  ;)18:01
besser82wiking: lot's of inlining and such....18:01
besser82wiking: with basically the same filenames, but different behaviour depending on what's defined....18:02
besser82wiking: that is what shreds the ccaches and make compiling that slow....18:03
besser82wiking: or (at least) recompiling...18:03
-!- Saurabh7 [~Saurabh7@] has joined #shogun18:07
@wikingbesser82: yes but u cannot simply run swig on those files alone18:07
@wikingbesser82: i.e. you cannot use ccache-swig18:07
-!- lisitsyn [~lisitsyn@] has joined #shogun18:17
-!- lisitsyn [~lisitsyn@] has quit [Quit: Leaving.]18:17
shogun-buildbot_build #2058 of deb3 - modular_interfaces is complete: Failure [failed examples and unit tests]  Build details are at  blamelist: Soeren Sonnenburg <>18:18
-!- zxtx [] has joined #shogun18:20
-!- zxtx [] has quit [Ping timeout: 245 seconds]18:26
-!- travis-ci [] has joined #shogun18:28
travis-ci[travis-ci] it's Soeren Sonnenburg's turn to pay the next round of drinks for the massacre he caused in shogun-toolbox/shogun:
-!- travis-ci [] has left #shogun []18:28
besser82wiking: I know... but there could be some possibility for a huge speed gain19:10
BoekeHi I got it working. I really appreciate your help and patience  with my low level of understanding. Thanks19:11
-!- thoralf_ [] has joined #shogun19:13
-!- thoralf [] has quit [Ping timeout: 246 seconds]19:16
-!- lisitsyn [~lisitsyn@] has joined #shogun19:20
-!- Saurabh7 [~Saurabh7@] has quit [Quit: Leaving]19:24
@wikingbesser82: again how?19:26
besser82wiking: I'm not sure,yet... trying to find a solution...19:27
-!- sonne|osx [] has joined #shogun19:32
sonne|osxlisitsyn: didn't help!19:36
-!- Saurabh7 [~Saurabh7@] has joined #shogun19:43
-!- thoralf [] has joined #shogun19:53
-!- thoralf_ [] has quit [Ping timeout: 245 seconds]19:56
-!- shogun-notifier- [] has quit [Quit: transmission timeout]20:29
@wikingbesser82: yeah the solution would be to do swig modules20:41
@wikingbut we had problems with that20:41
-!- lisitsyn [~lisitsyn@] has quit [Ping timeout: 245 seconds]21:45
-!- lisitsyn [~lisitsyn@] has joined #shogun21:48
-!- sonne|osx [] has quit [Quit: sonne|osx]22:23
-!- thoralf [] has quit [Quit: Konversation terminated!]22:37
-!- thoralf [] has joined #shogun22:38
-!- thoralf [] has quit [Client Quit]22:39
-!- thoralf_ [] has joined #shogun22:39
-!- thoralf_ [] has quit [Quit: Konversation terminated!]23:54
--- Log closed Wed Nov 27 00:00:42 2013