0

I have python code that acquires serial data from 2 devices and writes to a .txt file. Every 4-15 minutes there is approx 30-45 seconds of data missing in the .txt file and this is not acceptable for our use case. I've spent hours googling and searching SO about multiprocessing and serial port data acquisition and haven't come up with a solution.

Here is my code

gpsser = input(("Enter GPS comport as 'COM_': "))

ser = serial.Serial(port=gpsser,
                baudrate=38400,
                timeout=2,
                parity=serial.PARITY_NONE,
                stopbits=serial.STOPBITS_ONE,
                bytesize=serial.EIGHTBITS)

root = Tk()
root.title("DualEM DAQ")
path = filedialog.asksaveasfilename() + ".txt"
file = glob.glob(path)
filename = path
with open(filename, 'wb') as f:
    w = csv.writer(f, dialect='excel')
    w.writerow(['header'])


def sensor():
    while True:
        try:
            NMEA1 = dser.readline().decode("ascii")
            while dser.inWaiting() == 0:
                pass
            NMEA1_array = NMEA1.split(',')
            NMEA2_array = NMEA2.split(',')
            NMEA3_array = NMEA3.split(',')
            NMEA4_array = NMEA4.split(',')
            if NMEA1_array[0] == '$PDLGH':
                value1 = NMEA1_array[2]
                value2 = NMEA1_array[4]
            if NMEA1_array[0] == '$PDLG1':
                value3 = NMEA1_array[2]
                value4 = NMEA1_array[4]
            if NMEA1_array[0] == '$PDLG2':
                value5 = NMEA1_array[2]
                value6 = NMEA1_array[4]
                return (float(value1), float(value2), float(value3),
                        float(value4), float(value5), float(value6),
        except (IndexError, NameError, ValueError, UnicodeDecodeError):
            pass


def gps():
    while True:
        try:
            global Status, Latitude, Longitude, Speed, Truecourse, Date
            global GPSQuality, Satellites, HDOP, Elevation, Time
            while ser.inWaiting() == 0:
                pass
            msg = ser.readline()
            pNMEA = pynmea2.parse(msg)
            if isinstance(pNMEA, pynmea2.types.talker.RMC):
                Latitude = pynmea2.dm_to_sd(pNMEA.lat)
                Longitude = -(pynmea2.dm_to_sd(pNMEA.lon))
                Date = pNMEA.datestamp
            Time = datetime.datetime.now().time()
            if () is not None:
                return (Longitude, Latitude, Date, Time)
        except (ValueError, UnboundLocalError, NameError):
            pass

while True:
    try:
        with open(filename, "ab") as f:
                data = [(gps() + sensor())]
                writer = csv.writer(f, delimiter=",", dialect='excel')
                writer.writerows(data)
                f.flush()
        print(data)
    except (AttributeError, TypeError) as e:
        pass

The program is writing to the file but I need help understanding why I'm losing 30-45 seconds of data every so often. Where is my bottle neck that is causing this to happen?

Here is an example of where the breaks are, note the breaks are approx 50 seconds in this case.

Breaks in writing data to csv

DB

  • if () is not None: ---> what? why are there ser.flushInput; ser.flushOutput() ... ; inside? – Patrick Artner Apr 4 '18 at 21:43
  • 1. Thought process on if () is not none was see if the function returned a value to return that value and return nothing if it was none. I’m guessing I may not need that statement? 2.The ser.flushinput() is to clear the buffer of the incoming serial data. Does that make sense? – Dan B Apr 4 '18 at 23:32
  • Edited for updated code and added picture of time breaks for context. – Dan B Apr 5 '18 at 15:10
0

Back when I used PySerial, I did this:

nbytes = ser.inWaiting()
if nbytes > 0:
    indata = ser.read(nbytes)
    #now parse bytes in indata to look for delimiter, \n in your case
    #and if found process the input line(s) until delimiter not found
else:
    #no input yet, do other processing or allow other things to run
    #by using time.sleep()

Also note that new versions (3.0+) of PySerial have .in_waiting as a property not a method, so no (), it used to be .inWaiting().

| improve this answer | |
  • Try removing both while inWaiting() == 0 loops. The readline() calls should be blocking, all the inWaiting() loops are doing is consuming cpu until there is at least 1 byte available, after which your code will block in readline() until the rest of the line has arrived. – Cmaster Apr 5 '18 at 23:49
  • I've removed the inWaiting loops, however I'm still having the same issue. DB – Dan B Apr 6 '18 at 17:29
  • Run the gps and sensor loops in isolation to find out if the gap is due to one of those devices or the combination. – Cmaster Apr 7 '18 at 21:02
  • I ran the gps and sensor loops in isolation. Found a noisy powersupply to the gps reciever. Fixed that and it did reduce the problem but did not eliminate. So I ran the gps and sensor on different threads which did not help solve this problem either(I don't think it should've). Thanks for the idea on running the gps and sensor in isolation. I will continue to research what else may be going on – Dan B Apr 9 '18 at 19:21
0

You should not flush the serial port input. Data is arriving on its own timing into a buffer in the driver, not when your read happens, so you are throwing away data with your flush. You may need to add code to synchronize with the input stream.

| improve this answer | |
  • I updated the script removing the flushinput(). Still get same break in time after that. Do you have any advice, examples or ideas on how I should add code to syncronize with the input stream? – Dan B Apr 5 '18 at 15:29
0

I used threading with a queue and changed my mainloop to look like this.

while True:
  try:
    with open(filename, "ab") as f:
            writer = csv.writer(f, delimiter=",", dialect='excel')
            data = []
            data.extend(gpsdata())
            data.extend(dualemdata())
            writer.writerows([data])
            f.flush()
            f.close()
            dser.flushInput()
            ser.flushInput()
    print(data)
    sleep(0.05)
except (AttributeError, TypeError) as e:
    pass

I had to flush the serial port input data before the looping back to the read functions so it was reading new realtime data(this eliminated any lag of the incoming data stream). I've ran a 30 minute test and the time gaps appear to have go away. Thankyou to Cmaster for giving me some diagnostic ideas.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.