Open in new window / Try shogun cloud
--- Log opened Tue Oct 21 00:00:33 2014
-!- Phawx [~Phawx@91.102.9.98] has quit [Ping timeout: 245 seconds]00:26
-!- HeikoS [~heiko@nat-165-26.internal.eduroam.ucl.ac.uk] has quit [Quit: Leaving.]00:45
-!- Phawx [~Phawx@91.102.9.98] has joined #shogun00:47
-!- Phawx [~Phawx@91.102.9.98] has quit [Remote host closed the connection]01:06
-!- Floatingman [~Floatingm@c-68-52-34-232.hsd1.tn.comcast.net] has quit [Remote host closed the connection]06:58
-!- Floatingman [~Floatingm@c-68-52-34-232.hsd1.tn.comcast.net] has joined #shogun07:01
-!- PirosB3 [~pirosb3@195.101.111.125] has joined #shogun10:37
-!- HeikoS [~heiko@2.216.173.249] has joined #shogun11:25
-!- mode/#shogun [+o HeikoS] by ChanServ11:25
-!- Netsplit *.net <-> *.split quits: @lisitsyn13:37
-!- Netsplit over, joins: lisitsyn13:38
-!- mode/#shogun [+o lisitsyn] by ChanServ13:38
-!- PirosB3 [~pirosb3@195.101.111.125] has quit [Quit: PirosB3]14:41
-!- PirosB3 [~pirosb3@195.101.111.125] has joined #shogun14:50
-!- HeikoS [~heiko@2.216.173.249] has quit [Quit: Leaving.]16:20
-!- lambday [67157f4f@gateway/web/freenode/ip.103.21.127.79] has joined #shogun16:29
@lisitsynlambday: hey16:52
lambdaylisitsyn: hola16:52
@lisitsynlambday: how are you doing?16:52
lambdaylisitsyn: how do you do, sir?16:52
lambdaylisitsyn: haha.. good.. reading submodular functions is killing me16:53
@lisitsynlambday: hahah sir!16:53
-!- arne-cl [~arne@f052135204.adsl.alicedsl.de] has joined #shogun16:53
lambday(trying to add a British touch)16:53
@lisitsynlambday: before gsoc I receive like a billion of DEAR SIR16:53
@lisitsyn:D16:53
@lisitsynno offense but guess who writes this way :)16:53
lambdaylisitsyn: haha that's common.. I too got a few from anonymous people regarding asking tips for cracking the IIT entrance exam for masters :|16:54
@lisitsynlambday: did you manage to solve your issue with virtual stuff?16:54
lambdaylisitsyn: thanks to you.. I did - here it is16:54
lambdaylisitsyn: https://github.com/lambday/flash/blob/develop/src/flash/statistics/internals/DataManager.h#L11816:55
@lisitsynstd::shared_ptr<shogun::CFeatures>(feats)16:55
@lisitsynuhmm don't do that ;)16:55
@lisitsynmake_shared16:55
lambdaylisitsyn: ah that's just to save time16:55
lambdayohh16:55
lambdayalrighty16:55
@lisitsynahh no16:55
@lisitsynthis would be ok as well16:56
lambdaybut this doesn't leak16:56
@lisitsynyeah yeah my bad16:56
lambdaylisitsyn: why don't we start using shared_ptr in shogun as well whenever we can?16:56
@lisitsynlambday: I think we should16:57
@lisitsynlambday: we actually should convert to d-ptrs16:57
lambdaylisitsyn: and do we need a wrapper for the containers as well?16:57
@lisitsynsgvector?16:57
@lisitsynor what?16:57
@besser82lisitsyn, lambday: I'm currently accumulating stuff to cleanup ^^16:57
@lisitsynahh dynamicarray etc?16:57
lambdaynah normal std::vector std::pair std::map16:58
@lisitsynlambday: swig is slow with std16:58
@lisitsynthat's just the only issue16:58
@besser82lisitsyn, sgvector is custom impl. written in C  \o/16:58
@lisitsynbesser82: hah well there were some reasons I guess16:58
lambdaybesser82: also our CDynamicObjectArray is pretty heavyweight...16:58
lambdayotherwise it solves the same problems16:59
@besser82lisitsyn, yes --->  No invented here  XD16:59
@lisitsynI am like into training some models here now16:59
@lisitsynand actually for me what matters is command line :D16:59
@besser82lisitsyn, lambday: one thing is kick out all that legacy C*****-shiatz...16:59
@lisitsynso I start to think we should have cool command line for shogun16:59
@besser82that was just for those ugly $binding_static16:59
@besser82lisitsyn, real gangstaz sip on yacc  ^^17:00
@lisitsynbesser82: heh17:00
@lisitsynI mean VW is just an ugly thing with command line17:00
@lisitsynbut you can integrate it anywhere17:00
@besser82the whole cli-thingie is pretty ugly...17:01
lambdayI think that's a great idea..17:01
@lisitsynyeah cli is ugly17:01
@lisitsynbut writing huge programs for data processing is hard as well17:01
@lisitsynI mean you need to grep something sort it blabla and then feed to your classifier17:02
@lisitsynit is just a waste of time to write a program for that17:02
lambdaylisitsyn: how good is our cmdline_static?17:02
@besser82lisitsyn, that's what python_modular is for  ;)17:02
@lisitsynnah not really17:02
@lisitsynjust one CLI utility17:03
@lisitsynand your python data processing code is a mess :D17:03
arne-clhi! could someone help me compile/install shogun with python_modular bindings?17:03
@besser82lisitsyn, Y?17:03
@lisitsynarne-cl: yeah sure what's the problem?17:03
@besser82lisitsyn, import sh17:03
arne-cli tried foo@bar:/tmp/shogun-3.2.0/build# cmake -DCMAKE_INSTALL_PREFIX="/opt/shogun-install" -DPythonModular=ON .. which ran fine, but...17:03
lambdayarne-cl: hey17:03
@lisitsynbesser82: yeah but still ugly17:03
@besser82arne-cl, which distro?17:04
arne-cl... after make and make install, i cant import modshogun in python17:04
lambdayarne-cl: set PYTHONPATH17:04
arne-cllinux mint 1617:04
@lisitsynyeah17:04
@lisitsynyou have some strange prefix for shogun17:04
@lisitsynyou'd have to let python know of it17:04
@besser82arne-cl, export PYTHONPATH=/opt/shogun-install/usr/lib/pythonX.X/17:05
@besser82arne-cl, replace X.X with real python version ^^17:06
lambdaylisitsyn: did you happen to give any thoughts on the plugin arch?17:07
lambdayhow should we go about that... etc?17:07
@besser82lambday, I'd propose void**  XD17:08
lambdaybesser82: hahaha I C17:08
@lisitsynlambday: well still think17:09
@lisitsynwe should have a few interfaces17:09
@lisitsynlke base interfaces we have17:09
@lisitsynothers go away17:09
arne-clbesser82: i tried export PYTHONPATH=/opt/shogun-install/lib/python2.7/ but i still get an import error in python17:09
@lisitsynyou do some require - it loads .so (.dylib, .dll)17:10
lambdayso... the way I was thinking is to make the shogun code partitioned into wrappers and internals.. so in that case, the wrappers are the interfaces that you're talking about17:10
@lisitsynyeah17:10
@besser82arne-cl, you missed that **/usr/**  :D17:10
lambdayfits pretty well then..17:10
lambdayarne-cl: do a quick find :)17:10
arne-clbesser82: but in /opt/shogun-install, there are only these dirs: include, lib and share17:11
@lisitsynlambday: it is not clear for me how parameters should look like17:11
@lisitsynkernel.set("width", 3.0)?17:11
lambdaylisitsyn: in many cases I think we can make the internals header only... so highly optimized templated code can be written there..17:11
lambdayand wrappers are just clients using that internal library17:12
@lisitsynyeah whenever it makes sense we can optimize17:12
@besser82arne-cl, then I'm out of advice, sry...17:12
-!- Floatingman [~Floatingm@c-68-52-34-232.hsd1.tn.comcast.net] has quit [Read error: Connection reset by peer]17:12
lambdaylisitsyn: you did something for that in aer, no?17:12
@lisitsynlambday: yeah kind of17:12
arne-clbesser82: thanks anyway, i'll retry without a custom install dir17:13
lambdaylisitsyn: is there any off-the-shelf parameter framework that we can use?17:13
@lisitsynlambday: uhmm I don't know17:13
@lisitsynlambday: that's just about the interface17:13
@lisitsynstoring parameters is simple and stupid17:13
@lisitsynyou do some map or anything like that and that's it17:14
lambdaylisitsyn: so what's the issue?17:14
@lisitsynlambday: the interface17:14
lambdaykernel.set("width", 3.0) is fine17:14
lambdaypretty python-like :D17:14
@lisitsynlambday: the signature of such method is17:15
@lisitsynset(const char*, T)17:15
@lisitsynso we just generate all the possible things17:15
-!- Floatingman [~Floatingm@c-68-52-34-232.hsd1.tn.comcast.net] has joined #shogun17:15
@lisitsynI don't know if that's cool17:15
lambdayyeah17:15
@lisitsynno compile time checks at all17:15
lambdayummmm17:16
lambdayyeah that's an issue :/17:17
@lisitsynlambday: we can support both tags17:18
lambdaylisitsyn: so you don't want to do anything specific for each class?17:18
@lisitsynand strings17:18
@lisitsynthat's easy17:18
@lisitsynyeah no specific methods if we want plugins17:19
@lisitsynokay that's actually possible17:19
@lisitsynbut would be ugly17:19
@lisitsyncall('precompute_shit_together')17:19
@lisitsynlambday: ^ like that17:20
@lisitsynin python we can add things dynamically17:20
@lisitsynnot in java17:20
lambdayyou gotta have something for the classes to specify17:25
lambdaymake register_params mandatory for each class..17:26
@lisitsynlambday: I think we can avoid any register params17:26
@lisitsynlambday: you just don't specify field exactly17:26
@lisitsynyou only specify their names and type17:27
lambdayI mean yes that should do17:27
lambdayand you store as Any or void**17:27
lambdayand cast it back17:27
@lisitsynyeah Any and void** is the same :D17:27
@lisitsynlambday: you can do smart things if you know all the fields of a class17:28
lambdayso if the cast function is available then can't we static assert it with the templated set(...) function17:28
lambday:-/17:28
@lisitsyn1) you reserve exactly this amount of memory17:28
@lisitsyn2) you re-order them to be aligned17:28
@lisitsynand 3) you can add some spaces to avoid threading issues17:28
@lisitsynlambday: ^ 3) is the funny thing actually17:29
@lisitsynI am talking about false sharing17:29
lambdaylisitsyn: what's that?17:30
@lisitsynlambday: say you have17:30
@lisitsynstruct { int a; int b; }17:30
@lisitsynand you access it from different threads17:30
@lisitsynyour CPU will block both a and b just because they are in same cache line17:31
@lisitsynstruct { int a; char space[1024]; int b; } would be much faster :D17:31
lambdayreally17:31
@lisitsynlambday: so the thing is that ideally you'd just17:32
@lisitsynput some lines like that17:32
@lisitsynaddParameter<int>("width");17:32
@lisitsynaddParameter<std::string>("name");17:32
@lisitsynit reserves sizeof(int) + sizeof(std::string) + sizeof(alignment junk) bytes17:33
lambdaywhat if some single threaded program tries to use it then? locality is lost...17:33
@lisitsynlambday: yeah17:33
lambdayor it may be possible to use different layouts for single threaded or multithreaded if we know that too in advance17:34
@lisitsynlambday: I don't think we should bother about that17:34
@lisitsynlambday: the thing is that we can do that17:35
@lisitsynin one place17:35
@lisitsynif we have plain old C++ classes you know17:35
@lisitsynwe have to do all the stuff manually17:35
lambdaylisitsyn: how vital would be this alignment in some, say, data-oriented job?17:36
lambdaydata hungry17:36
@lisitsynnot really an issue I think17:36
@lisitsynlambday: we just let compiler know of constness of get17:37
@lisitsynand it optimizes redundant calls17:37
@lisitsynyou don't really use parameters that often I think17:37
lambdaylisitsyn: hmm... this is cool stuff man17:38
@lisitsynlambday: ahh yeah another thing about storing everything in one piece of memory17:39
@lisitsynlambday: we can (de)serialize with one strcpy17:39
@lisitsyn(if it is one arch)17:39
lambdaystrcpy or std::copy?17:40
@lisitsynlambday: what's the difference :D17:40
lambdaylol.. null :D17:40
@lisitsynlambday: pointers would be updated though17:43
lambdaylisitsyn: yep..17:45
lambdaylisitsyn: could we write some poc within aer for this stuff?17:45
@lisitsynlambday: yeah I'll try to find some time for that17:45
lambdaylisitsyn: if you think I can help code-wise for some places let me know..17:48
@lisitsynlambday: would assigning some issues work?17:48
lambdaylisitsyn: I guess so..17:48
@lisitsynlambday: cool then I'll try17:49
lambdaylisitsyn: awesome.. looking forward to this stuff.. very psyched17:49
@lisitsyn:D17:49
lambdaylisitsyn: reading all these mathematics is giving me headache :(17:49
lambdaycould use some break..17:49
@lisitsynlambday: oh why? maths is kind of cool17:49
lambdaylisitsyn: it is... but I feel like way too much new information is being hashed inside my brain within a very short period of time.. and I'm telling you I don't feel much good :D17:50
lambdaylisitsyn: working on feature selection btw which is cool stuff :D17:51
-!- arne-cl [~arne@f052135204.adsl.alicedsl.de] has left #shogun []17:51
lambdaylisitsyn: going to grab a beer... see you soon :)17:52
@lisitsynsee ya17:53
-!- PirosB3 [~pirosb3@195.101.111.125] has quit [Quit: PirosB3]19:13
@wikingping19:19
-!- lambday [67157f4f@gateway/web/freenode/ip.103.21.127.79] has quit [Ping timeout: 246 seconds]23:32
--- Log closed Wed Oct 22 00:00:35 2014