c# - Text reformatter gradually slows with each iteration -

-

edit 2

okay, i've posted copy of source @ gist.github , have 1 lingering problem can't resolve.

findline() returns -1. i've narrowed cause down if statement, can't figure out why. know symbol , symbollist both getting passed data.

/edit 2

i have simple c# program looks .csv file, reads text in file, reformats (and includes information sql query loaded datatable), , saves .tsv file later use program.

my problem source .csv file on 10,000 lines , program slows gradually iterates through lines. if .csv file ~500 lines takes 45 seconds complete, , time exponentially worse .csv file gets larger.

the sql query returns 37,000+ lines, requested once , sorted same way .csv file is, won't notice running through file unless can't find corresponding data, in case makes way through , returns appropriate error text. i'm 99% sure it's not cause of slowdown.

the y , z loops need long are.

if it's absolutely necessary can scrub data initial .csv file , post example, i'm hoping i'm missing obvious.

thanks in advance guys!

here's source:

using system; using system.collections.generic; using system.data; using system.data.sqlclient; using system.io; using system.linq; using system.text;  namespace moxysectorformatter {     class program     {         static void main(string[] args)         {             datatable resulttable = new datatable();             double curline = 1;             double numlines = 0;             string exportpath = @"***path***\***outfile***.tsv";             string importpath = @"***path***\***infile***.csv";             string newtext = "security\r\n";             string origtext = "";             string querystring = "select distinct upper(mp.symbol) symbol, lower(mp.sectype) sectype, mbi.status moxysecmaster mp left join moxybondinfo mbi on mp.symbol = mbi.symbol , mp.sectype = mbi.sectype mp.sectype <> 'caus' , mp.sectype not null , mp.symbol not null order symbol asc;";             sqlconnection moxyconn = new sqlconnection("server=***;database=***;user id=***;password=***");             sqldataadapter adapter = new sqldataadapter(querystring, moxyconn);              moxyconn.open();             console.write("importing source file \"{0}\".", importpath);             origtext = file.readalltext(importpath);             origtext = origtext.substring(origtext.indexof("\r\n", 0) + 2);             console.writeline("\rimporting source file \"{0}\".  done!", importpath);             console.write("scanning source report.");             (int loop = 0; loop < origtext.length; loop++)             {                 if (origtext[loop] == '\r')                     numlines++;             }             console.writeline("\rscanning source report.  done!");             console.write("downloading sectype information.");             resulttable = new datatable();             adapter.fill(resulttable);             moxyconn.close();             console.writeline("\rdownloading sectype information.  done!");              (int lcv = 0; lcv < numlines; lcv++)             {                 int foundspot = -1;                 int nextstart = 0;                 console.write("\rgenerating new file... {0} / {1} ({2}%)  ", curline, numlines, system.math.round(((curline / numlines) * 100), 2));                 (int vcl = 0; vcl < resulttable.rows.count; vcl++)                 {                     if (resulttable.rows[vcl][0].tostring() == origtext.substring(0, origtext.indexof(",", 0)).toupper() && resulttable.rows[vcl][1].tostring().length > 0)                     {                         foundspot = vcl;                         break;                     }                 }                 if (foundspot != -1 && foundspot < resulttable.rows.count)                 {                     newtext += resulttable.rows[foundspot][1].tostring();                     newtext += "\t";                     newtext += origtext.substring(nextstart, (origtext.indexof(",", nextstart) - nextstart));                     newtext += "\t";                     nextstart = origtext.indexof(",", nextstart) + 1;                     (int y = 0; y < 142; y++)                         newtext += "\t";                     if(resulttable.rows[foundspot][2].tostring() == "r")                         newtext += @"pre/etm";                     else if (origtext.substring(nextstart, (origtext.indexof(",", nextstart) - nextstart)) == "municipals")                     {                         newtext += "muni - ";                         nextstart = origtext.indexof(",", nextstart) + 1;                         if (origtext.substring(nextstart, (origtext.indexof(",", nextstart) - nextstart)).length > 0)                             newtext += origtext.substring(nextstart, (origtext.indexof(",", nextstart) - nextstart));                         else                             newtext += "(orphan)";                     }                     else if (origtext.substring(nextstart, (origtext.indexof(",", nextstart) - nextstart)) == "corporates")                     {                         newtext += "corporate - ";                         nextstart = origtext.indexof(",", nextstart) + 1;                         nextstart = origtext.indexof(",", nextstart) + 1;                         if (origtext.substring(nextstart, (origtext.indexof("\r\n", nextstart) - nextstart)).length > 0)                             newtext += origtext.substring(nextstart, (origtext.indexof("\r\n", nextstart) - nextstart));                         else                             newtext += "(unknown)";                     }                     else                         newtext += origtext.substring(nextstart, (origtext.indexof(",", nextstart) - nextstart));                     (int z = 0; z < 17; z++)                         newtext += "\t";                     newtext += "\r\n";                     resulttable.rows.removeat(foundspot);                 }                 else                     console.writeline("\r  omitting {0}: missing symbol or sectype.", origtext.substring(nextstart, (origtext.indexof(",", nextstart) - nextstart)));                 origtext = origtext.substring(origtext.indexof("\r\n", 0) + 2);                 curline++;             }             console.write("exporting file \"{0}\".", exportpath);             file.writealltext(exportpath, newtext);             console.writeline("\rexporting file \"{0}\".  done!\npress key exit.", exportpath);             console.readline();         }     } }

instead of using += operator concatenation, use system.text.stringbuilder object , it's append() , appendline methods

strings immutable in c#, every time use += in loop, new string created in memory, causing eventual slowdown.


Comments

Post a Comment

Popular posts from this blog

javascript - Enclosure Memory Copies -

-
function myclass() { //lots , lots of vars , code here. this.bar = function() { //do numerous enclosed myclass vars } this.foo = function() { alert('hello'); //don't enclosed variables. } } each instance of myclass gets own copy of bar , foo, why prototyped methods use less memory. however, know more memory use of inner methods. it seems obvious me (1) below must true. agree? not distinct myclass instances have own distinct copies of bar, must have own distinct copies of myclass enclosure. (or else how bar method keep myclass variables straight on per instance basis?) now (2) question i'm after. since inner foo method doesn't use in myclass enclosure natural qustion is: javascript smart enough not keep myclass enclosure in memory use of foo? this depend entirely on implementation not language. naive implementations keep far more around others. answer true e.g. v8 (chrome's js engine) may not true spider...
Read more

php - Replacing tags in braces, even nested tags, with regex -

-
example preg_replace('/\{[a-za-z.,\(\)0-9]+\}/', 'replaced', 'lorem ipsum dolor sit {tag1({tag2()})}, consectetur adipiscing elit.'); the result: lorem ipsum dolor sit {tag1(replaced)}, consectetur adipiscing elit. question as can see "tag2" has been replaced, want replace "tag1" know how can this? (in cases might this: {tag1({tag2({tag3()})})}) , on.) btw using preg_replace_callback, easier show preg_replace here site can test code: http://www.spaweditor.com/scripts/regex/index.php you need add curly braces character set. here's pattern used: /\{[a-za-z.,\(\)\{\}0-9]+\}/ and here result: "lorem ipsum dolor sit replaced, consectetur adipiscing elit."
Read more