Open in new window / Try shogun cloud
--- Log opened Sat May 26 00:00:41 2012
-!- n4nd0 [] has quit [Ping timeout: 265 seconds]00:13
-!- n4nd0 [] has joined #shogun00:29
-!- n4nd0 [] has quit [Client Quit]00:29
shogun-buildbotbuild #573 of csharp_modular is complete: Success [build successful]  Build details are at
-!- blackburn [~blackburn@] has quit [Ping timeout: 245 seconds]02:35
-!- blackburn [~blackburn@] has joined #shogun02:39
-!- blackburn [~blackburn@] has quit [Ping timeout: 246 seconds]03:40
-!- pluskid [~pluskid@] has joined #shogun07:37
-!- n4nd0 [] has joined #shogun09:22
CIA-113shogun: Jacob Walker master * r349d66c / (5 files in 2 dirs): Added Element-wise Product Kernel. This Kernel is based heavily on -
CIA-113shogun: Soeren Sonnenburg master * rf29facf / (5 files in 2 dirs): Merge pull request #555 from puffin444/master -
CIA-113shogun: Soeren Sonnenburg master * r156a436 / (6 files): use appropriate label types for ruby modular -
shogun-buildbotbuild #796 of octave_static is complete: Failure [failed test_1]  Build details are at  blamelist: walke434@msu.edu09:43
shogun-buildbotbuild #797 of octave_static is complete: Success [build successful]  Build details are at
-!- n4nd0 [] has quit [Ping timeout: 246 seconds]10:37
-!- n4nd0 [] has joined #shogun11:04
-!- pluskid [~pluskid@] has quit [Ping timeout: 246 seconds]11:04
-!- pluskid [] has joined #shogun11:04
n4nd0hi pluskid, I have just read your mail11:06
n4nd0I really like the idea11:07
pluskidn4nd0: :D11:08
pluskidI remembered that I saw somebody trying to set up standard ml benchmarks that compares some state-of-the-art algorithms on some *real world* datasets, but no longer be able to find this site, maybe he didn't continue...11:09
n4nd0pluskid: interesting11:09
n4nd0pluskid: let's try not to forget about it and maybe we can think of it more carefully once our GSoC projects are more advanced11:10
n4nd0and well, let's see what the others think :)11:10
-!- pluskid [] has quit [Ping timeout: 246 seconds]12:15
-!- pluskid [~pluskid@] has joined #shogun12:33
-!- blackburn [~blackburn@] has joined #shogun13:06
-!- n4nd0 [] has quit [Quit: leaving]13:54
-!- pluskid [~pluskid@] has quit [Ping timeout: 246 seconds]13:58
-!- pluskid [] has joined #shogun13:58
-!- blackburn [~blackburn@] has quit [Ping timeout: 250 seconds]15:41
-!- puffin444 [62e3926e@gateway/web/freenode/ip.] has joined #shogun16:00
-!- oliver [55b43dc2@gateway/web/freenode/ip.] has joined #shogun16:01
oliverpuffin444, just ping me when you are here.16:02
puffin444oliver: I'm here16:02
oliverPerhaps let's start with what we'd like to discuss.16:03
oliverAnything specific on your side?16:03
oliverI mainly a have a number of suggestions of order and how to avoid the (usual) pain in getting some of the GP details to work to save your time.16:04
oliverBut if you want to start with sth. else - let's do that first.16:04
puffin444I do not have any question right now16:05
oliverOk. What is on your list for next week?16:05
puffin444I believe I start working on the inference methods according to the plan16:06
-!- Francis_Chan1 [~Adium@] has joined #shogun16:07
olivergreat. That's a good time then to discuss a few things.16:07
oliverYou previously coded up the basic GP model.16:07
oliverI'd just like to share a few points of things which could be cumbersome to get moving quickly.16:07
oliverAs a first goal I'd use the simlest possible GP model and basically just extend your existing GP class.16:08
oliverYou have a flexible kernel function in there already; the only thin you need is to add noise, a diagonal term.16:08
oliverHow did you plan to handle the noise in the standard Gaussian case?16:08
puffin444Do you mean the eta in y = f(x) + eta ?16:09
oliverTHere is some ambiguity. For standard GP stuff noise is effectively just a component of the covariance - which is one option.16:09
oliverI mean: is it strictly part of the likelihood model?16:09
oliverPerhaps the cleanest way to start out with16:09
oliverBasically: what I'd suggest is to just get the GP class you have now running, including gradient optimization and simple Gaussian noise.16:10
oliverJust one kernel (RBF would be suitable), Gaussian noise and gradient based optimization.16:11
oliverWhen you have that start structuring it into different components, it's merely pulling the pieces apart.16:11
oliverHave you looked into an interface for kernels to provide gradient information and math needed to calc the derivatives of marginal likelihoods w.r.t. kernel parameters etc. ?16:12
puffin444No I actually have not. Right now I only have the likelihood model return derivatives. Perhaps I need to add this interface16:14
oliverYes, you will need it one way or the other.16:14
oliverHave you done stuff with gradient optimization before ?16:15
puffin444Only in simple cases. I wrote a gradient search for a neural network a while back.16:16
oliverok, great.16:16
oliverWhat's generally good here.16:16
oliverYou have a complex function, which is the marginal likelihood16:16
oliverThere is a likelihood model and the kernel.16:16
oliverTHe kernel may itself be a product or sum of other kernels.16:16
oliverIn the end you want the gradient and it better be correct.16:16
oliverIt's best to check gradients for every component independently using a grad check - did you use dthat back then?16:17
oliverIt's part of most optimizers, but the approach is merely to compute the numerical gradient and compare to the analytical form.16:17
oliveri.e. f'(x) = (f(x+h)-f(x)) / h16:18
oliveryou just calc that for small h16:18
oliverif multi dimensional as here, you have one step in every direction.16:18
oliverthen you can check that this numericla implementation of f' matches your analytical solution.16:18
oliveryou can debug that for the kernel separately and all components and then put together.16:18
puffin444Okay. Because we are computing the gradient analytically, doesn't this mean that an specific way to compute the gradient must be added to every kernel?16:19
oliverevery kernel needs to calculate the derivate of all kernel entries w.r.t. to parameters16:19
oliverI'd start with RBF kernel first16:19
oliverdo you have matlab?16:19
puffin444I have octave16:20
puffin444and I can get access to matlab if necessary16:20
oliveryeah, equally good. YOu can either steal from gpml as you wrote in your proposal or if you prefer python I have a similar thing in pythohn.16:20
oliverBoth implement all these gradients for various kernels, soo good to get inspiration of how to do things.16:20
oliverFOr example, here are implementations of the rbf kernel derivaives:16:22
oliverThis contains a variant of grad check to get inspiration from:16:22
oliverAnd this is an example of a simple demo with optimization we should try to replicate in shogun (very similar ones in gpml)16:23
oliverI guess take a look and ping me if you need anything. It will take you a bit of time to get it to work at all (and getting the maths right), so keep things low complexity with few classes until you have a working thing and then build all the framework machinery on top.16:24
oliverI am travelling quite a bit next week, so best is to arrange discussions via email.16:25
oliverI'll be back to regular office mode on Firday16:25
puffin444So in this case I should first work with the class I have (GPRegression) and write what is necessary to learn the hyperparameters?16:26
puffin444After this is accomplished, write the infrastructure for this functionality to be generalized16:26
oliverexactly. Play around with a simple test bed where you have everything under control and can move quickly.16:27
oliverAnd then complicate it.16:27
puffin444What do you think is a reasonable timeframe for this? According to may project plan all the infrastructure needed for hyperparameter learning16:27
puffin444(no approximations or non-gaussian likelihoods) is written by half time16:28
oliverI think if you strip down all the complexity.16:28
oliverTake just one kernel, one likelihood model and the GP base class that coud be 1-2 weeks.16:29
oliverThere is a lot of code (python/matlab) to get inspiration from.16:29
puffin444I was thinking the same16:29
oliverProbably it's more on the order of 2 weeks.16:29
oliverThen you have always a version to compare to and can check that the framework etc. is correct and produces the correct results.16:30
puffin444In 1-2 weeks, get the simple case running correctly, and then the next two weeks generalize this into the infrastructure we discussed earlier16:30
oliveryes, agree.16:30
oliverAnd use gradcheck for everything ;-)16:31
oliverI never coded up a single gradient that was correct first shot.16:31
puffin444How should I go about merging this into the main branch? Should I just make changes to my fork and only have a pull request at the very end?16:33
oliverI think it's good to get all the feedback early on, otherwise it will be painful in the end.16:33
oliverIf it's clearly code that is temporary keep it local, otherwise merging is good.16:33
oliverEven if some of the GP base class will change it ensures that the code is in line with shogun guide lines right from the start.16:34
oliverAnything else?16:36
puffin444So I will make pull request incrementally16:36
puffin444Not at this point16:36
puffin444Thanks for taking time for this meeting16:36
oliverLet's keep in touch. Email will be best next week.16:37
oliverNo prob, thanks for getting up early. I am just leaving tonight and thought it would be good to catchup before then.16:37
oliverTalk soon.16:38
puffin444I will probably be emailing you over next week.16:38
puffin444Do you have any other questions for me?16:39
oliverNot right now.16:39
oliverLet's use email till Friday. Take care and happy coding.16:40
puffin444Okay. You too.16:40
-!- oliver [55b43dc2@gateway/web/freenode/ip.] has quit [Quit: Page closed]16:40
-!- puffin444 [62e3926e@gateway/web/freenode/ip.] has left #shogun []16:42
-!- Francis_Chan1 [~Adium@] has left #shogun []16:50
-!- pluskid [] has quit [Quit: Leaving]18:57
-!- gsomix [~gsomix@] has joined #shogun19:01
gsomixhi all19:01
-!- blackburn [~blackburn@] has joined #shogun19:40
-!- gsomix [~gsomix@] has quit [Ping timeout: 244 seconds]19:46
-!- blackburn [~blackburn@] has quit [Quit: Leaving.]19:46
-!- blackburn [~blackburn@] has joined #shogun19:46
-!- gsomix [~gsomix@] has joined #shogun19:59
@sonney2kgsomix, hi there ... so how is the director kernel stuff progressing?20:04
gsomixsonney2k, moin, I'm working now.20:05
gsomixjust checked directors with templates, trying integrate it to shogun20:06
CIA-113shogun: Soeren Sonnenburg master * r216b658 / examples/undocumented/ruby_modular/classifier_libsvm_minimal_modular.rb : fix ruby example -
gsomixsonney2k, and about exams. I have last at 4 June (there are some exams after, but it's easy).20:13
gsomixthat's all20:13
blackburnthat's all folks :D20:16
@sonney2kgsomix, ok... but what about the simple test I suggested?20:18
@sonney2kgsomix, enabling directors for some *simple* class in shogun that derives from SGObject?20:18
@sonney2kgsomix, I mean even just an example like the one you had20:20
@sonney2kyou could just take this class and derive it from sgobject20:20
@sonney2kthen test if it still works within all the shogun stuff20:20
* sonney2k awaits the buildbot to complete with 100% success now20:21
gsomixsonney2k, =___= please wait, I need 15-20 mins.20:22
shogun-buildbotbuild #799 of octave_static is complete: Failure [failed test_1]  Build details are at  blamelist: sonne@debian.org20:33
gsomixsonney2k, directored DynamicArray works fine.20:44
gsomixbut there is just simple overloaded setter.20:45
gsomixbut with templates20:45
gsomixsonney2k, should I commit it?20:47
gsomixsimple test20:48
* gsomix afk20:50
@sonney2kgsomix, thats very cool then :))21:19
@sonney2kthen I really wonder why my directorkernel didn't work...21:19
@sonney2kgsomix, maybe because it was a protected method!21:21
blackburnsonney2k: did you try to redefine compute?21:21
@sonney2kHow could I miss that compute() of CKernel is protected21:22
* sonney2k smacks head21:22
* sonney2k once21:22
* sonney2k twice21:22
* sonney2k bumps head on the table21:22
* sonney2k ouuuuch21:22
gsomixsonney2k, yep. I think problem in protected methods.21:29
gsomixsonney2k, what to do next?21:30
CIA-113shogun: Soeren Sonnenburg master * r817fb76 / src/shogun/kernel/DirectorKernel.h : create *public* kernel_function method to be overidden in directorkernel -
@sonney2kgsomix, please have a look at this21:33
@sonney2kin principle it should now be possible to override kernel_function when you compile shogun with --enable-swig-directors21:34
@sonney2kthe exampe for that is in kernel_director_modular.py21:35
@sonney2kit needs some adjustments - maybe you create an example your own21:35
gsomixsonney2k, ok21:43
gsomixsonney2k, do you have some ideas for directored kernel?21:44
@sonney2kgsomix, maybe you do the reverselinear kernel from the shogun tutorial in python21:44
@sonney2kgsomix, in libshogun examples kernel_revlin.cpp21:45
blackburnmay be just compute linear21:45
blackburnand compare matrices21:45
@sonney2kblackburn, or that21:46
blackburngsomix: linear kernel is numpy dot21:46
@sonney2kblackburn, one has to pay attention though21:46
@sonney2klinear kernel might be normalized21:46
blackburnto accuracy21:46
blackburnnormalization ture21:46
* sonney2k checks21:46
blackburnno need to check before21:47
blackburngsomix: just generate random data and compare linear kernel with director kernel21:48
blackburncompute is21:48
@sonney2kblackburn, no it uses identity kernel normalizer21:48
@sonney2kso all good21:48
blackburnjust numpy.dot21:48
@sonney2kblackburn, it is not so easy21:48
@sonney2kthere is sth we should discuss21:48
@sonney2koperates on indices21:49
@sonney2kso how should it work21:49
@sonney2kshould one do get_lhs()->get_feature_vector(row)21:49
blackburnlhs = get_lhs().get_computed_feature_vector()21:49
@sonney2ksame for rhs21:49
@sonney2kand then dot()21:49
blackburnor lhs = X[:,i]21:49
blackburnwhere X is a feature matrix21:49
@sonney2kor should the features be set somewhere in the overloaded class?21:49
@sonney2kblackburn, well...21:50
@sonney2kif data is in the class21:50
blackburnlet me write compute function21:50
@sonney2kthen we need a way to set num_lhs / num_rhs in directorkernel21:50
@sonney2kotherwise kernel.kernel(i,j) will check if i<num_lhs21:50
blackburnI think no need to have extra features21:51
@sonney2kand fail because num_lhs is 021:51
@sonney2kgsomix, please add two functions to directorkernel21:51
blackburnsonney2k: whY?21:51
@sonney2kvoid set_num_vec_lhs(int32_t num_lhs)21:52
@sonney2kand set_num_vec_rhs21:52
blackburnI think no need to add extra data to class21:52
@sonney2kblackburn, you might have some complex data you want to compute kernels for21:52
@sonney2kblackburn, think of graphs21:52
@sonney2kor parse trees21:52
@sonney2kblackburn, we have one issue here21:53
@sonney2kif we use directors we cannot use threads21:53
blackburnokay fits perfectly with openmping21:54
@sonney2kso we have to make sure that whenever some director* stuff is involved that num_threads = 121:54
@sonney2kblackburn, no21:54
@sonney2ksame problem21:54
@sonney2kpython has the GIL21:54
blackburnwe have threads only in kernel matrix21:54
blackburnlet me check21:54
@sonney2kso calling the (from python) overloaded function multiple times in parallel from threads is not possible21:54
blackburnokay just some virtual function for # of threads21:55
@sonney2kgsomix, so please when you do the test do kernel.parallel.set_num_threads(1)21:55
@sonney2kgsomix, then kernel.get_kernel_matrix()21:55
blackburnI have no idea when to continue with openmp21:56
blackburnI need someone to help me to test it21:56
@sonney2kblackburn, openmp is not the issue here21:56
@sonney2kproblem will be there with openmp nevertheless21:56
blackburnyes yes I understand21:56
blackburnjust recalled21:56
@sonney2kgsomix, so once you have implemented the linear director kernel as example21:57
@sonney2kjust compute it's kernel matrix21:57
@sonney2kand compare how fast computing the matrix is (say for a 1000x1000) matrix21:57
@sonney2kcompared to using LinearKernel() from shogun21:57
blackburnfast heh21:57
@sonney2kI would expect about 10-100 times slower...21:57
blackburndo you care?21:58
blackburndifferent thing21:58
gsomixsonney2k, ok21:59
@sonney2kI care yes. That will tell us the overhead and to what extend this is at all useful21:59
@sonney2kif it is 1000 times slower it is not worth the effort22:00
blackburnit is worth anyway I believe22:00
@sonney2kgsomix, doing all that should take just 1-2 hours...22:00
blackburntoo cool to leave22:01
@sonney2kcool indeed22:01
shogun-buildbotbuild #161 of nightly_default is complete: Success [build successful]  Build details are at
gsomixsonney2k, ok, time to work.22:04
blackburnhmm lets try to kill some doc warinngs22:04
gsomixbtw, today I worked as builder.
gsomixI think I should add some text about shogun and gsoc at this photo...22:04
@sonney2khardcore talent abuse!22:06
blackburnsonney2k: typical russian house :D22:07
blackburnsonney2k: that's what I think about your stl hate:
gsomixblackburn, pink >__<22:14
blackburnyes yes my gf's room is pink22:14
gsomixI don't have photos with me and blackburn :(22:18
shogun-buildbotbuild #800 of octave_static is complete: Success [build successful]  Build details are at
CIA-113shogun: Sergey Lisitsyn master * r676b942 / (3 files in 3 dirs): Removed a few doc warnings -
shogun-buildbotbuild #558 of ruby_modular is complete: Success [build successful]  Build details are at
gsomixsonney2k, have strange error23:27
gsomixTraceback (most recent call last):23:27
gsomix  File "", line 43, in <module>23:27
gsomix    kernel_director_linear_modular(*parameter_list[0])23:27
gsomix  File "", line 31, in kernel_director_linear_modular23:27
gsomix    dkernel.init(feats_train, feats_train)23:27
gsomixRuntimeError: maximum recursion depth exceeded while calling a Python object23:27
gsomixand init...23:28
gsomixvirtual bool init(CFeatures* l, CFeatures* r)23:28
gsomixreturn CKernel::init(l, r);23:28
gsomixin DirectorKernel23:28
blackburnwhat is code?23:28
gsomixand I know, that my kernel_function is wrong23:30
gsomixtired, need to sleep23:30
blackburnhmm try to trace calls23:30
blackburnwhat is calling by recursive?23:30
blackburninit calls init?23:31
gsomixblackburn, yep in Kernel23:32
gsomixgood night, guys =___=23:33
-!- gsomix [~gsomix@] has quit [Ping timeout: 244 seconds]23:39
@sonney2kthat is the error I was getting23:53
blackburnyes I do remember23:55
--- Log closed Sun May 27 00:00:41 2012