Open in new window / Try shogun cloud
--- Log opened Mon Jul 27 00:00:10 2015
-!- shaochuan [] has joined #shogun01:53
-!- shaochuan [] has quit [Ping timeout: 252 seconds]01:58
-!- HeikoS [~heiko@] has joined #shogun02:01
-!- mode/#shogun [+o HeikoS] by ChanServ02:01
-!- HeikoS [~heiko@] has quit [Quit: Leaving.]02:15
-!- PirosB3 [] has quit [Quit: PirosB3]02:57
-!- shaochuan [~shaochuan@2601:647:4600:fac5:e97f:35a0:bfa0:fc28] has joined #shogun06:19
-!- shaochuan [~shaochuan@2601:647:4600:fac5:e97f:35a0:bfa0:fc28] has quit [Ping timeout: 246 seconds]06:26
-!- PirosB3 [] has joined #shogun10:02
-!- shaochuan [] has joined #shogun10:04
-!- shaochuan [] has quit [Ping timeout: 264 seconds]10:09
-!- PirosB3 [] has quit [Quit: PirosB3]10:36
-!- HeikoS [~heiko@] has joined #shogun11:06
-!- mode/#shogun [+o HeikoS] by ChanServ11:06
-!- ajph [~ajph@unaffiliated/ajph] has left #shogun ["Textual IRC Client:"]11:26
lisitsynHeikoS: hey12:28
lisitsynstammtisch today rye?12:28
-!- HeikoS [~heiko@] has quit [Quit: Leaving.]12:39
-!- shaochuan [~shaochuan@2601:647:4600:fac5:e97f:35a0:bfa0:fc28] has joined #shogun13:06
-!- shaochuan [~shaochuan@2601:647:4600:fac5:e97f:35a0:bfa0:fc28] has quit [Ping timeout: 244 seconds]13:11
-!- xwize [892c01ae@gateway/web/freenode/ip.] has joined #shogun13:53
xwizewhy do I need svn to cmake this!?!?!? ngraargh13:57
@wikingu dont need svn at all to cmake anything :)14:01
xwizeI'm on windows and I need to build an x64 version, I've never used this before, I tried cmake . and after a while it complains about svn checkout of MSIntTypes14:03
@wikingatm shogun will not compile on windows natively14:03
@wikingu r better of compiling that on linux14:04
xwizebut I'm doing all my dev on windows, which is my native OS, and I can't virtualise because I'm using the GPU and I need full performance14:05
@wikingmmm sorry then :(14:05
@wikingthat svn stuff was a first attmept to get shogun compiled with visual studio... but it's not even near to be done14:06
xwizeI was looking forward to using this! I guess I should find something else14:07
@wikingyeah or help in porting it to windows :)14:07
xwizesorry, I'd like to help but I'm sort of in a hurry and my own project is giving me grey hairs14:11
@wikingyeah sorry again windows natively isn't supported atm14:13
-!- HeikoS [] has joined #shogun14:33
-!- mode/#shogun [+o HeikoS] by ChanServ14:33
-!- HeikoS1 [~heiko@] has joined #shogun15:32
-!- HeikoS [] has quit [Ping timeout: 244 seconds]15:32
-!- xwize [892c01ae@gateway/web/freenode/ip.] has quit [Quit: Page closed]15:33
-!- HeikoS1 [~heiko@] has quit [Quit: Leaving.]15:40
-!- HeikoS [~heiko@] has joined #shogun15:40
-!- mode/#shogun [+o HeikoS] by ChanServ15:40
-!- HeikoS [~heiko@] has quit [Ping timeout: 265 seconds]15:53
-!- PirosB3 [] has joined #shogun16:40
-!- HeikoS [] has joined #shogun17:10
-!- mode/#shogun [+o HeikoS] by ChanServ17:10
-!- shogun-notifier- [] has joined #shogun17:25
shogun-notifier-shogun: Sergey Lisitsyn :develop * f00e611 / CMakeLists.txt:
shogun-notifier-shogun: Remove inline-functions option17:25
shogun-notifier-shogun: Sergey Lisitsyn :develop * 5bd5d07 / CMakeLists.txt:
shogun-notifier-shogun: Merge pull request #2868 from lisitsyn/bugfix/yosemite_clang17:25
shogun-notifier-shogun: Remove inline-functions option17:25
-!- PirosB3 [] has quit [Quit: PirosB3]17:27
lisitsynHeikoS: hey17:28
@HeikoSlisitsyn: hey there17:28
lisitsynHeikoS: I glanced over optimizers PRs17:28
lisitsynI have no idea :D17:28
lisitsynI have a feeling it has too much pointer juggling17:29
lisitsynand code could be reduced17:29
@HeikoSlisitsyn: I was asking you because you have a good idea of how to build elegant c++ code17:30
@HeikoSwhat wu is doing is a bit crazy17:30
@HeikoSlike 5 new classes17:30
@HeikoSfor things like cost functions17:30
lisitsynHeikoS: ok we need faster chat because I constantly forget commenting PRs17:30
lisitsynHeikoS: one approach I find quite good is17:30
lisitsynto write test/example first17:31
lisitsynand then write the api to satisfy it17:31
lisitsynwe need a few use cases17:31
lisitsynwe have a function blabla, want to minimize it17:32
@HeikoSI see17:32
lisitsynI think this could be better17:32
@HeikoSThe thing is that Wu wants things to be modular17:32
@HeikoSand I agree on that17:32
@HeikoSso his approach is just taking the OOP way17:32
@HeikoSbut all these objects onherit from CSGobject17:32
@HeikoSso we get like 800 lines of code per class17:33
lisitsynOOP is not the best way you know17:33
lisitsynsometimes we just need a function17:33
@HeikoSthis is what I had in mind17:33
lisitsynthere is no need for interface of the function we minimize17:33
@HeikoScould you saw a few things on this in the PR?17:33
lisitsynHeikoS: yeah but I am not sure how should we turn wu into the right way17:34
lisitsynI mean we risk to overcriticize :)17:34
lisitsynHeikoS: do you have good understanding what's the goal?17:35
@HeikoSGPs have different objective functions17:35
@HeikoSand there are different way to optimise17:35
@HeikoSso he wants to do that from a base class17:35
@HeikoSbut I feel that function pointers should do things here17:35
@HeikoSlisitsyn: he can take critic17:36
@HeikoSlisitsyn: the thing is I cannot merge these giant PRs17:36
@HeikoSI have no idea17:36
lisitsynI have a suggestion17:36
lisitsynto not use shogun things for this optimizers17:36
lisitsynat all17:36
lisitsynI mean use simple (maybe even C) api17:36
lisitsynwhat do you think?17:36
-!- travis-ci [] has joined #shogun17:36
travis-ciit's Sergey Lisitsyn's turn to pay the next round of drinks for the massacre he caused in shogun-toolbox/shogun:
-!- travis-ci [] has left #shogun []17:36
lisitsynI fell in love with C api17:37
lisitsynit is much cleaner just always :D17:37
lisitsynI would even write tapkee in C today17:37
shogun-buildbotbuild #30 of FC22 - libshogun is complete: Failure [failed test]  Build details are at  blamelist: Sergey Lisitsyn <>17:38
@HeikoSlisitsyn: depends on the details17:38
@HeikoSlisitsyn: could you have a look in the PR?17:38
shogun-buildbotbuild #1045 of FCRH - libshogun is complete: Failure [failed test]  Build details are at  blamelist: Sergey Lisitsyn <>17:38
@HeikoSI also feel Wu should probably draw a class diagram to discuss things17:38
lisitsynok let me comment17:38
shogun-buildbotbuild #2710 of bsd1 - libshogun is complete: Failure [failed test]  Build details are at  blamelist: Sergey Lisitsyn <>17:40
shogun-buildbotbuild #2667 of deb3 - modular_interfaces is complete: Failure [failed csharp modular]  Build details are at  blamelist: Sergey Lisitsyn <>17:40
lisitsynHeikoS: ok commented17:43
shogun-buildbotbuild #653 of deb4 - python3 is complete: Failure [failed test python modular]  Build details are at  blamelist: Sergey Lisitsyn <>17:47
@wikingfuck i wont be able to be awake again for stammtisch17:52
lisitsynwiking: ae17:59
-!- PirosB3 [] has joined #shogun18:21
-!- PirosB3 [] has quit [Client Quit]18:21
-!- yorkerlin [b8af2f1e@gateway/web/freenode/ip.] has joined #shogun18:31
yorkerlinlisitsyn, please take a look at the comments I wrote at
lisitsynyorkerlin: hey18:33
lisitsynyorkerlin: ok I have a suggestion how can we simplify design18:35
lisitsynwhat if we write the code that uses that 'library'18:35
lisitsynI mean how do you see it being used18:35
yorkerlinI think it is a tool for ML developers18:37
lisitsyndo you see it as a standalone thing?18:37
yorkerlinwhat do you mean standalone thing?18:37
lisitsynI mean if you write that thing so it can be compiled without shogun18:38
lisitsynit could be better18:38
lisitsynpeople could take it and use18:38
-!- PirosB3 [] has joined #shogun18:38
lisitsynand you don't have to bother with our interface oriented things like our own map and sgvector18:38
yorkerlinic, should I use std lib for vector?18:39
lisitsynyorkerlin: no I mean you would be free to use whatever you want to18:40
yorkerlinyes. if we do not use sgvector, the framework can be complied without shogun18:41
yorkerlinand cmap18:41
lisitsynyorkerlin: there is also a cool technique to write C++ code under C api18:42
lisitsynit speeds up compilation and make it usable from say python18:42
lisitsynI don't suggest just telling you variants18:42
yorkerlinany reference18:42
lisitsynok let me find18:42
yorkerlinlisitsyn, did see the comments about learning rate and descenupdater? (function pointers seem to be not enough)18:45
lisitsynyorkerlin: hmm didn't find it yet, ok let me send you one thing18:45
lisitsynyorkerlin: do you feel like glancing over a book?18:45
lisitsynI have been reading a cool book on C++ API18:46
lisitsynjust a sec18:46
yorkerlinAPI Design for C++?18:46
yorkerlinok, I have a copy of the book.18:47
lisitsynok :)18:48
lisitsynok lets get back to this thing then18:48
lisitsynI have got another idea meanwhile18:49
lisitsynI have seen some pretty design here at my job18:49
lisitsynso if we have some optimizer18:49
lisitsynit could be good idea to split mutable and immutable parts18:49
yorkerlinhmm, good suggestion18:50
lisitsynyorkerlin: so you could create some class that contains all state of optimizer18:50
lisitsyncurrent pointer, gradient18:50
lisitsynerr sorry18:50
lisitsynpointer=point :)18:50
yorkerlinstate, u mean the mutable part?18:52
lisitsynbasically anything you need to stop and resume18:52
lisitsynso that other parts are just a matter of configuration18:52
yorkerlinyes. one issue is learning rate, gradient update, penalty have many choices.18:54
lisitsynok as I can see18:55
lisitsynLR, gradient, penalty just modify this mutable state18:55
lisitsynso we can just make them a functions over mutable state18:56
lisitsyna set of functions18:56
lisitsynit would be less clutter I think18:56
lisitsyninterfaces are ok but they bloat code a bit18:57
lisitsynin java they are the only way18:57
lisitsynbut here we can make it shorter18:57
yorkerlinso these set of functions should share the same function definition, right?18:58
yorkerlinfor example void fun1(int a,int b), void fun2(int a, int b)18:58
lisitsynjust a sec18:59
lisitsynhmm I am not sure now19:00
lisitsynprobably functions are not that good because we can mix them up19:00
yorkerlinlet me give an example19:00
yorkerlinfor inverse scaling learn rate, learning_rate = eta0 / pow(t, power_t), where t is the times we call get_learning_rate(...), eta0 and power_t must be given by users or default value.19:00
yorkerlinfor const learning rate, learning_rate=const_learning_rate, const_learning_rate is given by users or default value19:01
lisitsynyorkerlin: something like that?19:02
yorkerlinok. I am look at it19:02
lisitsynyorkerlin: I mean optimizer would be pretty generic19:04
lisitsynI'll be around later, like in a hour19:05
lisitsyngoing home :)19:05
yorkerlinok see then19:05
lisitsynsee you a bit later19:05
-!- shaochuan [~shaochuan@2601:647:4600:fac5:e97f:35a0:bfa0:fc28] has joined #shogun19:11
-!- shaochuan [~shaochuan@2601:647:4600:fac5:e97f:35a0:bfa0:fc28] has quit [Ping timeout: 246 seconds]19:15
@HeikoSyorkerlin: hi!20:02
@HeikoSyorkerlin: good to catch you, how are things?20:04
yorkerlingood. lisitsyn suggested another way to implement the framework.20:04
@HeikoSyorkerlin: just reading through the logs20:05
@HeikoSyorkerlin: do you see why I am a bit concerned about the current structure?20:05
yorkerlinbecause of  CSGObject ?20:06
@HeikoSyorkerlin: yeah thats one thing20:06
@HeikoSyorkerlin: whenever we add a new class that inherits from CSGObject, we get a few hundred lines of code to compile20:07
@HeikoSyorkerlin: and all the overhead from the base class that we dont need20:07
@HeikoSyorkerlin: only objects that are directly exposed via the interfaces should inherit from this class20:07
yorkerlinone issue is how to deal with serialization20:08
@HeikoSyorkerlin: the other thing is the sheer number of classes. The OOP approach here is a bit convoluted in my eyes20:08
yorkerlinif we do not use CSGObject as the base class20:08
@HeikoSyorkerlin: how do you mean?20:08
@HeikoSyorkerlin: can use CSGObject for the main class, and make it have pointers to the data where needed.20:08
@HeikoSyorkerlin: but things like cost functions do not need to be serialised, or?20:09
yorkerlinI am not sure how the  serialization work in shogun. I think CSGObject take care of the serialization work.20:09
@HeikoSyorkerlin: yes thats what it does20:10
yorkerlinthat is why I use the CSGObject as the base class20:10
@HeikoSyorkerlin: btw I agree that it is bad to have global variables20:11
@HeikoSyorkerlin: just want to avoid the number of classes exploding20:11
yorkerlinI think lisitsyn suggested another way
@HeikoSyorkerlin: yeah this seems cleaner, question is: would it work here?20:12
@HeikoSIt seems like this would allow to flatten out the code a bit20:13
yorkerlinyes. I think so. however, there will be many classes (eg, l2penlaty, const_learning_rate, gradient_updater)20:13
yorkerlinif we want to combine them20:13
@HeikoSyorkerlin: if they are not extending CSGObject, all this is fine20:14
@HeikoSyorkerlin: they are only used internally right?20:14
@HeikoSnot from say Python20:14
@HeikoSits just for the GP code developers20:14
yorkerlinin fact we can do this
yorkerlinnot just for gp20:15
@HeikoSyorkerlin: agreed!20:15
@HeikoSon my todo list for long time ;)20:15
@HeikoSyorkerlin: so but back to the classes20:15
@HeikoSif you need a state, you can just define a simple class20:15
@HeikoSno difference to c-structs in fact, these things almost look the same in memory20:16
@HeikoSaprt from v-tables if you have purely virtual methods20:16
@HeikoSso you can do these things as long as they are not exposed via serialisation or Python/SWIG20:16
yorkerlinabout the serialization thing. maybe we can document how the serialization works20:17
@HeikoSand you can use constructors and init methods of the CSGObject-based classes to set up the right combination of loss, etc20:17
@HeikoSyorkerlin: sure20:17
@HeikoSyorkerlin: Though I have the intention to drop this anyways20:17
@HeikoSyorkerlin: but: it works like this:20:17
@HeikoSyorkerlin: CSGObject has two methods for this: save_serializable, and load_serializable20:17
@HeikoSAll parameters that were added by SG_ADD macro are serilaised20:18
@HeikoSeasy as that20:18
@HeikoSthats why you have to register parameters20:18
@HeikoSthey methods themselves look horrible and contain a lot of code20:18
@HeikoSthat is why adding a new CSGObject class to Shogun increases compile time quite abit20:18
yorkerlinso if we do not use CSGObject as base class, when dserializable, we need to new the instance. right?20:19
@HeikoSyorkerlin: how do you mean?20:19
yorkerlinhow does the deSerialization work20:20
@HeikoSwe can only de-serialise CSGObject files20:20
@HeikoSthe way it works is:20:20
@HeikoSempty instance is created (init method is called, so class knows all the pointers)20:20
@HeikoSand load_serializable allocates and populates the memory of its parameters20:20
@HeikoSyorkerlin: it is pretty horrible to be honest. I am for dropping serialisation20:21
yorkerlinlet me give an example for inverse scaling learn rate, learning_rate = eta0 / pow(t, power_t), where t is the times we call get_learning_rate(...), eta0 and power_t must be given by users or default value.20:21
@HeikoSyorkerlin: in my MCMC code (python) learning rates are functions of t20:21
@HeikoSyorkerlin: and the main class calls them with increasing values of t20:21
yorkerlinhowever, there are many learning rate methods.20:22
@HeikoSbut I see, you mean there are paramters given by users20:22
yorkerlinconst learning rate, line search methods,20:22
yorkerlininverse scaling learning rate20:22
@HeikoSand they all have different parameters, so we cannot just use an enum20:22
@HeikoSbecause then the main class would have to store all parameters20:23
@HeikoSI see20:23
yorkerlinthe problem is if a new learning rate method is added, we may need to modify the main class :(20:23
@HeikoSyorkerlin: agreed that is not good20:23
@HeikoSyorkerlin: but CSGObject as base class is neither a good idea20:24
@HeikoSyorkerlin: will users change the type of learning rate?20:24
@HeikoSyorkerlin: how many learning rates will there be?20:24
@HeikoSyorkerlin: in many places in Shogun, such situations are solved via an enum and a switch in the main class20:25
yorkerlinusers (I mean, developer) have to init a learning rate instance20:25
@HeikoSsee QuadraticTimeMMD20:25
-!- shogun-notifier- [] has quit [Quit: transmission timeout]20:25
yorkerlinthe issue is learning rate may contain mutable variables.20:26
@HeikoSI see20:26
yorkerlinif we dp serialization and do deserialization20:26
@HeikoSyorkerlin: question is: how many learning rates will there be?20:26
@HeikoSyorkerlin: and do they have different numbers of parameters?20:26
@HeikoSif there are just 3-4, and then it is unlikely to add another one, these can go into the main class20:27
@HeikoSyorkerlin: its not like a kernel, where there are hundreds of possibilities20:27
yorkerlinif we do not add linear search methods (for batch minimizers), there are 4 learning rate methods20:28
@HeikoShow many parameters do they have?20:28
@HeikoShow many more could there be?20:28
yorkerlinif we add line search methods,20:28
yorkerlinat least 3 more methods20:28
@HeikoSyorkerlin: but all of them are one-liners as functions, not?20:28
@HeikoSyorkerlin: what are the mutable parts of the learning rate?20:29
@HeikoSiteration number is not really part of learning rate for example20:29
yorkerlin for inverse scaling learn rate, learning_rate = eta0 / pow(t, power_t),20:29
yorkerlint is the mutable20:29
yorkerlineta0, power_t are also mutable I think20:30
@HeikoSbut can't t be provided by the caller?20:30
yorkerlinyou need store t in the caller20:31
@HeikoSt is a parameter to any learning rate, isnt it?20:31
@HeikoSso then why not a static function where the eta0 and the power_t parameters are fixed?20:31
@HeikoSlambda t: eta0/t**power_t20:32
@HeikoSand eta0 was defined before20:32
yorkerlinfor SGD, it is possible. for GD, I am not sure yet20:33
@HeikoSyorkerlin: what would be a problem there?20:33
yorkerlinIn line search methods for GD or batch minimizer, we may need to try guess the learning rate20:35
@HeikoSyorkerlin: where would that guessing happen and based on which information?20:35
yorkerlingradient and variable from the cost function20:36
@HeikoSbut thats part of another class then20:36
@HeikoSnot part of the learning rate, so needs to be passed20:37
@HeikoSyorkerlin: but then again, one can go back to a static function as all information is passed by the caller20:37
@HeikoSand the caller needs to know the parameters of the learning rate anyways (otherwise he cannot modify)20:38
yorkerlinyes. again, if a new method is proposed, the static function  may be changed.20:38
@HeikoSso that breaks the modular structure anyways as two leaf classes need to know each others instance, no?20:38
yorkerlinif we are ok fine with that20:38
-!- PirosB3 [] has quit [Quit: PirosB3]20:38
@HeikoSyorkerlin: my optinion is that this doesnt happen often, so I would go with that.20:39
@HeikoSyorkerlin: but:20:39
@HeikoSit's your code, I just wanted to have a discussion on this20:39
@HeikoSlisitsyn should also give his opinion here20:39
@HeikoSjust to make sure these things are consciously decided20:40
yorkerlinanother thing is momentum update20:40
@HeikoSyorkerlin: details?20:41
yorkerlinfor momentum update, we need to store a corrected_gradient. for plain update, we do not need to store  the correct_gradient20:41
yorkerlinif we go futher using inexact Hessian information, we may need to store the information20:42
yorkerlinI mean the caller cannot store all these information20:43
@HeikoSyorkerlin: if the learning rate class were to store that, where would it get it from?20:43
yorkerlingradientupater should store the needed information20:43
yorkerlinoh, momentum update is not learning rate20:44
@HeikoSyorkerlin: I am confused20:44
yorkerlinI agree with you that learning rate can be use some static functions20:44
@HeikoSyorkerlin: btw: do you have an idea how other strongly typed optimisation toolboxes solve these issues?20:45
yorkerlinlet me find some20:46
@HeikoSyorkerlin: just asking, since I don't20:46
@HeikoSthe gradient information in the momentum are only helper variables right?20:47
@HeikoSI mean they do not need to be visible to the outside20:47
@HeikoSand they do not need to be serialised20:47
@HeikoSso I guess we could just use classes that dont inherit from CSGObject20:47
yorkerlinhowever, the gradient information in the momentum is mutable20:47
@HeikoSyorkerlin: sure, but that is only read/written during the algorithm20:48
@HeikoSso no serialisation happening then, right?20:48
@HeikoSyorkerlin: I gotta go now, good talking to you20:49
yorkerlinfor increamental inference, may be not20:49
@HeikoSyorkerlin: maybe have another chat with sergey20:49
@HeikoSand lambday20:49
@HeikoShe is really good in these things20:49
@HeikoSmaybe send him an email, he might not have seen the discussion in the PR20:49
@HeikoSyorkerlin: Im back tomorrow, we could chat then again20:50
yorkerlinthe issue all come form "serilization and then  deserilization"20:50
@HeikoSyorkerlin: keep in mind we might drop it20:50
@HeikoSyorkerlin: but yeah you are right20:51
@HeikoSyorkerlin: lets just avoid to inherit from CSGObject if possible. Only the main interfaced classes need to20:51
@HeikoSyorkerlin: take care, bye20:51
yorkerlinfor incremental inferce, we first do minimizer for existing data.we  then do serilization. in some point, we do  deserilization and do incremtnal update.20:52
yorkerlinwe want the mutable variables when we do the incremental update.20:53
yorkerlinwe may called warm_start  at
-!- HeikoS [] has quit [Ping timeout: 244 seconds]20:56
yorkerlinhi, I leave now. will send email to your guys21:20
-!- yorkerlin [b8af2f1e@gateway/web/freenode/ip.] has quit [Quit: Page closed]21:20
-!- PirosB3 [] has joined #shogun22:32
-!- PirosB3 [] has quit [Client Quit]22:36
-!- thoralf [] has joined #shogun22:50
-!- mode/#shogun [+o thoralf] by ChanServ22:50
-!- thoralf [] has quit [Quit: Konversation terminated!]23:02
--- Log closed Tue Jul 28 00:00:11 2015