python - Subprocess Popen invalid argument/broken pipe while communicating to C program -
i have code
all needed libraries imported
class vertex(structure): _fields_ = [("index", c_int), ("x", c_float), ("y", c_float)] other stuff
this create , array list of vertex
def writelist_buf(size, nomeid): nvert_vertex_array_type = vertex * len(bpy.data.objects[nomeid].data.vertices) passarr = nvert_vertex_array_type() in range(len(passarr)): vert = bpy.data.objects[nomeid].data.vertices[i] passarr[i] = vertex(vert.index, vert.co[0], vert.co[1]) return passarr bpy.data.objects[nomeid].data.vertices list of vertices.
other stuff
this inside def, , communicate c program previous array
input = writelist_buf(size, nomeid) c_program_and_args = "here program arguments(it works)" cproc = popen(c_program_and_args, stdin=pipe, stdout=pipe) out, err = cproc.communicate(input) #the program returns 2 integers separed space return [int(i) in out.decode.split()] size , nomeid declared before writelist call.
after bit of "debugging" found type passed writelist_buf "legal"(it's bytes, since array created c_types), keep receiving errno32 broken pipe or errno22 invalid argument... c program make read in stdiin retrive vertices(like c code below)..
the strange think before "integrating" inside code working on, have tried simpler code: one, , works!
from subprocess import popen, pipe ctypes import * class vertex(structure): _fields_ = [("index", c_int), ("x", c_float), ("y", c_float)] nverts = 5 vlist = [vertex(0,1,1), vertex(1,2,2), vertex(2,3,3), vertex(3,4,4), vertex(4,5,5)] array = vertex * nverts input = array() in range(nverts): input[i] = vlist[i] print(type(input)) cproc = popen("pipeinout.exe random arg", stdin=pipe, stdout=pipe) out, err = cproc.communicate(input) print(out.decode()) and c code
#include<stdio.h> #include<stdlib.h> typedef struct { int index; float x; float y; } vertex; int main(int argc, char* argv[]) { int n=5; int i; printf("%s",argv[1]); vertex* vv; vv=(vertex*)malloc(sizeof(vertex)*n); fread(vv,sizeof(vertex),n,stdin); //fread(&vv,sizeof(vv),1,stdin);//metti nel valore di vv(non quello che punta) l'indirizzo passato||sizeof(vv) size of pointer for(i=0;i<n;i++) printf(" %i , %f , %f\n",vv[i].index,vv[i].x,vv[i].y); }
from comments understand pass millions of items hundreds of times c program. approach below (pipe input using subprocess) might slow in case. possible alternatives write c extension (e.g., using cython) or use ctypes call c functions directly. ask separate question describing use case in detail approach preferable.
if you've chosen approach make sure works correctly before optimization (write tests, measure performance , after optimize if needed) -- make work, make right, make fast.
on other hand there no point invest time in approaches known thrown away later -- fail fast.
if output of c program bounded; .communicate() method code works (source):
import struct, sys subprocess import popen, pipe vertex_struct = struct.struct('i f f') def pack(vertices, n): yield struct.pack('i', n) v in vertices: yield vertex_struct.pack(*v) def main(): try: n = int(sys.argv[1]) except indexerror: n = 100 vertices = ((i,i+1,i+2) in range(n)) p = popen(["./echo_vertices", "random", "arg"], stdin=pipe, stdout=pipe) out, _ = p.communicate(b''.join(pack(vertices, n))) index, x, y = vertex_struct.unpack(out) assert index == (n-1) , int(x) == n , int(y) == (n+1) if __name__ == '__main__': main() here's the code comments question. works without errors large n values on machine:
import struct, sys subprocess import popen, pipe threading import thread def pack(vertices, n): yield struct.pack('i', n) s = struct.struct('i f f') v in vertices: yield s.pack(*v) def write(output_file, chunks): chunk in chunks: output_file.write(chunk) output_file.close() def main(): try: n = int(sys.argv[1]) except indexerror: n = 100 vertices = ((i,i+1,i+2) in range(n)) p = popen(["./echo_vertices", "random", "arg"], stdin=pipe, stdout=pipe) thread(target=write, args=[p.stdin, pack(vertices, n)]).start() line in iter(p.stdout.readline, b''): pass p.stdout.close() sys.stdout.buffer.write(line) p.wait() if __name__ == '__main__': main() q&a
q: don't understand pack functions (i know yield returns iterable object iterable 1 time, in code use 2 yield, don't returns.
pack() generator. generators not work how you've described them, e.g.:
>>> def f(): ... yield 1 ... yield 2 ... >>> in f(): ... print(i) ... 1 2 note each yield produces value.
>>> def g(n): ... in range(n): ... yield ... >>> list(g(10)) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] here's yield present in text 1 time executed 10 times , each time produces value (an integer in case). see generators in python tutorial. "generator tricks systems programmers" contains multiple examples on how use generators simple advanced usage.
q: in addition dont know (*v) means @ line 10)
s.pack(*v) calls pack method using argument unpacking:
>>> def h(a, b): ... print(a, b) ... >>> h(*[1, 'a']) 1 >>> h(*range(2)) 0 1 >>> h(0, 1) 0 1 q: don't how thread in line 25 works,
thread(target=write, args=[p.stdin, pack(vertices, n)]).start() this line starts new thread calls write() function arguments args keyword argument i.e. output_file=p.stdin , chunks=pack(vertices, n). write() function in case equivalent to:
p.stdin.write(struct.pack('i', n)) p.stdin.write(s.pack(0, 1, 2)) p.stdin.write(s.pack(1, 2, 3)) ... p.stdin.write(s.pack(n-1, n, n+1)) p.stdin.close() after thread exits.
q: ...and read output of program.. isn't stored in variable, it?
the whole output not stored anywhere. code:
for line in iter(p.stdout.readline, b''): pass reads p.stdout line-by-line until .readline() returns empty string b'' , stores current line in line variable (see iter() docs). so:
sys.stdout.buffer.write(line) just prints last line of output.
q: 1)after starting thread, python script waits until finished, right?
no, main thread exits. started thread not daemon. runs until completes i.e., script (the program) doesn't exit until completes.
q: 2)i understood how read stdout of c program,but don't when start it.afa understood,with write function write in buffer(or file in ram) data want,and when run c program, can read data wrote.but when start c program in code? :)
the c program started p = popen(...).
p.stdin.write() writes stdin of c program (there number of buffers in between can forget moment). process same in:
$ echo abc | some_program q: 3)last thing: why use wait on p? there's warning http://docs.python.org/library/subprocess.html?#subprocess.popen.wait
for provided c code not necessary write p.stdin in separate thread. use thread avoid situation described in warning i.e., c program produces enough output before script finishes writing stdin (your c code doesn't write before finishes reading thread not necessary).
in other words p.wait() safe in case.
without p.wait() stderr output c program might lost. though can reproduce stderr loss on jython the scripts. yet again provided c code doesn't matter due not writing stderr anything.
Comments
Post a Comment