map - Having two sets of input combined on hadoop -

-

i have rather simple hadoop question i'll try present example

say have list of strings , large file , want each mapper process piece of file , 1 of strings in grep program.

how supposed that? under impression number of mappers result of inputsplits produced. run subsequent jobs, 1 each string, seems kinda... messy?

edit: not trying build grep map reduce version. used example of having 2 different inputs mapper. let's lists , b , mapper work on 1 element list , 1 element list b

so given problem experiences no data dependency result in need chaining jobs, option somehow share of list on mappers , input 1 element of list b each mapper?

what trying built type of prefixed look-up structure data. have giant text , set of strings. process has strong memory bottleneck, therefore after 1 chunk of text/1 string per mapper

mappers should able work independent , w/o side effects. parallelism can be, mapper tries match line patterns. each input processed once!

otherwise multiply each input line number of patterns. process each line single pattern. , run reducer afterwards. chainmapper solution of choice here. remember: line appear twice, if matches 2 patterns. want?

in opinion should prefer first scenario: each mapper processes line independently , checks against known patterns.

hint: can distribute patterns distributedcache feature mappers! ;-) input should splitted inputlineformat


Comments

Post a Comment

Popular posts from this blog

javascript - Enclosure Memory Copies -

-
function myclass() { //lots , lots of vars , code here. this.bar = function() { //do numerous enclosed myclass vars } this.foo = function() { alert('hello'); //don't enclosed variables. } } each instance of myclass gets own copy of bar , foo, why prototyped methods use less memory. however, know more memory use of inner methods. it seems obvious me (1) below must true. agree? not distinct myclass instances have own distinct copies of bar, must have own distinct copies of myclass enclosure. (or else how bar method keep myclass variables straight on per instance basis?) now (2) question i'm after. since inner foo method doesn't use in myclass enclosure natural qustion is: javascript smart enough not keep myclass enclosure in memory use of foo? this depend entirely on implementation not language. naive implementations keep far more around others. answer true e.g. v8 (chrome's js engine) may not true spider...
Read more

php - Replacing tags in braces, even nested tags, with regex -

-
example preg_replace('/\{[a-za-z.,\(\)0-9]+\}/', 'replaced', 'lorem ipsum dolor sit {tag1({tag2()})}, consectetur adipiscing elit.'); the result: lorem ipsum dolor sit {tag1(replaced)}, consectetur adipiscing elit. question as can see "tag2" has been replaced, want replace "tag1" know how can this? (in cases might this: {tag1({tag2({tag3()})})}) , on.) btw using preg_replace_callback, easier show preg_replace here site can test code: http://www.spaweditor.com/scripts/regex/index.php you need add curly braces character set. here's pattern used: /\{[a-za-z.,\(\)\{\}0-9]+\}/ and here result: "lorem ipsum dolor sit replaced, consectetur adipiscing elit."
Read more