Skip to content

Conversation

@ddipp
Copy link

@ddipp ddipp commented May 11, 2019

Sometimes it is required to get the values of variables from gnuplot. For example after:

gnuplot> f(x) = m*x + b
gnuplot> fit f(x) 'calibration.data' using 1:2 via m, b
iter      chisq       delta/lim  lambda   m             b            
   0 1.5671569869e+02   0.00e+00  4.44e+00    1.000000e+00   1.000000e+00
   1 1.6202293571e+00  -9.57e+07  4.44e-01    4.193938e-01   8.336373e-01
   2 2.1262262773e-02  -7.52e+07  4.44e-02    4.900177e-01   1.305259e-01
   3 1.2002761375e-02  -7.71e+05  4.44e-03    4.994356e-01   6.485437e-02
   4 1.2002753199e-02  -6.81e-01  4.44e-04    4.994444e-01   6.479260e-02
iter      chisq       delta/lim  lambda   m             b            

After 4 iterations the fit converged.
final sum of squares of residuals : 0.0120028
rel. change during last iteration : -6.81204e-07

degrees of freedom    (FIT_NDF)                        : 8
rms of residuals      (FIT_STDFIT) = sqrt(WSSR/ndf)    : 0.0387343
variance of residuals (reduced chisquare) = WSSR/ndf   : 0.00150034

Final set of parameters            Asymptotic Standard Error
=======================            ==========================
m               = 0.499444         +/- 0.004265     (0.8538%)
b               = 0.0647926        +/- 0.02646      (40.84%)

correlation matrix of the fit parameters:
                m      b      
m               1.000 
b              -0.886  1.000 
gnuplot> print m_err
0.00426450345234437
gnuplot> print b_err
0.0264605480528866
gnuplot> 

Now from Python you can access these variables.

pi = gp.get('pi')
Returns a string
@benschneider
Copy link
Owner

benschneider commented May 12, 2019

Thank you very much for the contribution!
This is indeed quite good and would give access to the superior Gnuplot fit function.

The problem comes with the "Popen.communicate()" function, as it terminates the pipe after usage: https://stackoverflow.com/questions/25754045/python-subprocess-i-o-operation-on-closed-file
So I am looking for a way to avoid it, and allow multiple readouts.

@ddipp
Copy link
Author

ddipp commented May 12, 2019

Thank you very much for the contribution!
This is indeed quite good and would give access to the superior Gnuplot fit function.

The problem comes with the "Popen.communicate()" function, as it terminates the pipe after usage: https://stackoverflow.com/questions/25754045/python-subprocess-i-o-operation-on-closed-file
So I am looking for a way to avoid it.

Oh, I'm sorry, I hurried.
What if you rewrite it like this:
Library

import sys
from subprocess import PIPE, Popen
from threading import Thread

try:
    from queue import Queue, Empty
except ImportError:
    from Queue import Queue, Empty  # python 2.x

ON_POSIX = 'posix' in sys.builtin_module_names


def enqueue_std(out, queue):
    for line in iter(out.readline, ''):
        queue.put(line)
    out.close()


class Gnuplot(object):
    """docstring for Gnuplot"""

    def __init__(self):
        self.p = Popen(['gnuplot', '-p'], stdin=PIPE, stderr=PIPE, stdout=PIPE, bufsize=1,
                       close_fds=ON_POSIX, shell=False, universal_newlines=True)
        self.q_err = Queue()
        self.t_err = Thread(target=enqueue_std, args=(self.p.stderr, self.q_err))
        self.t_err.daemon = True  # thread dies with the program
        self.t_err.start()

        self.q_out = Queue()
        self.t_out = Thread(target=enqueue_std, args=(self.p.stdout, self.q_out))
        self.t_out.daemon = True  # thread dies with the program
        self.t_out.start()

    def c(self, command):
        self.p.stdin.write(command + '\n')  # \n 'send return in python 2.7'
        self.p.stdin.flush()  # send the command in python 3.4+

    def get(self, variable, timeout=0.1):
        self.p.stdin.write('print ' + variable + '\n')
        # read line without blocking
        try:
            line = self.q_err.get(timeout=timeout)  # or .get_nowait()
            return line
        except Empty:
            return None

Use library:

#!/usr/bin/env python3

from gnuplot import Gnuplot as gp

figure1 = gp()

figure1.c('set terminal postscript eps color enhanced "Helvetica" 30')
figure1.c('set output "%s"' % 'result.ps')
figure1.c('set size 1.25,1.33')
figure1.c('set pointsize 2')
figure1.c('set nokey')
figure1.c('set linestyle 1 linetype -1 linewidth 1')
figure1.c('set xlabel "Detector"')
figure1.c('set ylabel "Calibration Quantity"')
figure1.c('set title "Title"')
figure1.c('plot sin(x)')

my_pi = figure1.get('pi')
print(my_pi)

And:

./test.py 
3.14159265358979
ls
gnuplot.py  __pycache__  test.py  result.ps

@benschneider
Copy link
Owner

Thanks again,

This is a good find, and seems to work.

In fact, the communicate function also seems use a thread for the readout
https://github.com/python/cpython/blob/master/Lib/subprocess.py

I will experiment with the options a little and see what works best.

@benschneider
Copy link
Owner

I will use your suggested code since it is very clean and works very well!
Thank you very much Dmitry, excellent work!

@ddipp
Copy link
Author

ddipp commented May 12, 2019

I made small edits.
When you request the value of a variable, there may be data in the buffer. They need to reset.
Also, after sending the command to gnuplot, you may need to read the errors or the log.

import sys
from subprocess import PIPE, Popen
from threading import Thread

try:
    from queue import Queue, Empty
except ImportError:
    from Queue import Queue, Empty  # python 2.x

ON_POSIX = 'posix' in sys.builtin_module_names


class Gnuplot(object):
    """docstring for Gnuplot"""

    def __init__(self):
        self.p = Popen(['gnuplot', '-p'], stdin=PIPE, stderr=PIPE, stdout=PIPE, bufsize=1,
                       close_fds=ON_POSIX, shell=False, universal_newlines=True)
        self.q_err = Queue()
        self.t_err = Thread(target=self.enqueue_std, args=(self.p.stderr, self.q_err))
        self.t_err.daemon = True  # thread dies with the program
        self.t_err.start()

        self.q_out = Queue()
        self.t_out = Thread(target=self.enqueue_std, args=(self.p.stdout, self.q_out))
        self.t_out.daemon = True  # thread dies with the program
        self.t_out.start()

    def enqueue_std(self, out, queue):
        for line in iter(out.readline, ''):
            queue.put(line)
        out.close()

    def c(self, command):
        self.p.stdin.write(command + '\n')  # \n 'send return in python 2.7'
        self.p.stdin.flush()  # send the command in python 3.4+

    def get_c(self, vtype=str, timeout=0.1):
        # read line without blocking
        lines = []
        while True:
            try:
                line = self.q_err.get(timeout=timeout)  # or .get_nowait()
                lines.append(vtype(line.strip()))
            except Empty:
                break
        return lines

    def get(self, variable, vtype=float, timeout=0.1):
        # clean buffer before put command
        self.get_c()
        self.p.stdin.write('print ' + variable + '\n')
        # read line without blocking
        try:
            line = self.q_err.get(timeout=timeout)  # or .get_nowait()
            return vtype(line.strip())
        except Empty:
            return None

use:

#!/usr/bin/env python3

from gnuplot import Gnuplot as gp

figure1 = gp()

figure1.c('set terminal postscript eps color enhanced "Helvetica" 30')
figure1.c('set output "%s"' % 'result.ps')
figure1.c('set size 1.25,1.33')
figure1.c('set pointsize 2')
figure1.c('set nokey')
figure1.c('set linestyle 1 linetype -1 linewidth 1')
figure1.c('set xlabel "Detector"')
figure1.c('set ylabel "Calibration Quantity"')
figure1.c('set title "Title"')
figure1.c('set fit logfile "/dev/null"')
figure1.c('set fit quiet')
figure1.c('FIT_LIMIT = 1e-6')
figure1.c('f(x) = m*x**2 + b')
figure1.c('fit f(x) "calibration.data" using 1:2 via m, b')
figure1.c('plot "calibration.data" using 1:2, f(x) w l')

# !!! wrong command !!!
figure1.c('wrong command')

log = figure1.get_c()
print("\n".join(log))

m = figure1.get('m')
b = figure1.get('b')
m_err = figure1.get('m_err')
b_err = figure1.get('b_err')
print("m = {0}".format(m))
print("b = {0}".format(b))
print("m_err = {0}".format(m_err))
print("b_err = {0}".format(b_err))
./regression.py 

gnuplot> wrong command
^
line 0: invalid command

m = 0.04317673009566
b = 1.14943292650526
m_err = 0.00344378125924026
b_err = 0.173332048570009

I do not know how to check the result of the execution of each command. The timeout is now used, since the execution of a command in gnuplot may take some time. If you check after each command, it will be very slow.

@ddipp
Copy link
Author

ddipp commented May 12, 2019

I also found it very strange: gnuplot does not use stdout. It sends the entire message (data output or error messages) to strerr!
It is very uncomfortable. Cannot separate messages and errors.

@ddipp
Copy link
Author

ddipp commented May 12, 2019

Wow! Just need to read the documentation.

gnuplot> help set print
 The `set print` command redirects the output of the `print` command to a file.

 Syntax:
       set print
       set print "-"
       set print "<filename>" [append]
       set print "|<shell_command>"
       set print $datablock [append]

 `set print` with no parameters restores output to <STDERR>.  The <filename>
 "-" means <STDOUT>. The `append` flag causes the file to be opened in append
 mode.  A <filename> starting with "|" is opened as a pipe to the
 <shell_command> on platforms that support piping.

 The destination for `print` commands can also be a named data block. Data
 block names start with '$', see also `inline data`.

To separate messages into errors and data, you need to say in the object's constructor:

self.c('set print "-"')

@benschneider
Copy link
Owner

Thats actually explains a lot, also why my earlier attempts to just read stdout failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants